Ancient and old book cover image retrieval method based on dynamic partitioning and multi-modal feature fusion

Abstract

The invention discloses an old and ancient book cover image retrieval method based on dynamic partitioning and multi-modal feature fusion, and relates to the technical field of image recognition, old and ancient book cover images are used as query vectors to be input into a retrieval model, and a matched book list corresponding to old and ancient books is obtained according to the query vectors; the training process of the retrieval model is as follows: performing visual feature extraction and text feature extraction on the cover image of the ancient and old book, and partitioning; inputting the blocked visual features and text features into a cross attention mechanism, and generating an original high-dimensional fusion feature vector including a layout text structure and image layout information; performing subspace decomposition and quantization coding on the original high-dimensional fusion feature vector by adopting a product quantization algorithm to obtain a compressed feature piece vector, and constructing an ancient and old book feature database according to the compressed feature piece vector; taking cover images of the old and ancient books as input query vectors, indexing the old and ancient book database by adopting a three-level progressive query architecture, and returning a matched book list; the trainable parameters of the retrieval model are jointly optimized by constructing a total loss function; according to the retrieval method, multi-level understanding and digital reconstruction of ancient book covers are achieved, and rapid retrieval is achieved through efficient index construction.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method for retrieving images of old book covers based on dynamic segmentation and multimodal feature fusion. Background Technology

[0002] Antique books, as an important historical and cultural heritage, carry rich historical, cultural, and academic value. However, due to their age and limited preservation conditions, many antique books suffer from problems such as damaged covers, blurred text, yellowed or soiled paper, which poses many challenges to their digital management and retrieval.

[0003] Currently, the retrieval of ancient books mainly relies on text-based methods, which extract textual information from books using OCR (Optical Character Recognition) technology and then search using keyword matching. However, this method has the following limitations: Low OCR recognition accuracy: The text in ancient books may have faded or blurred due to age, or use special printed fonts, handwritten fonts, or even variant characters, resulting in a high OCR recognition error rate and affecting retrieval accuracy. Version diversity issues: The same book may have multiple versions published in different periods and regions, with different cover designs, layout styles, and printing quality, making it difficult to distinguish different versions based solely on textual content. Underutilized cover information: Many ancient book covers contain unique visual features (such as prints, seals, binding styles, etc.), but existing retrieval systems usually ignore this information, relying only on metadata such as book title and author for matching, resulting in inaccurate search results. Insufficient retrieval efficiency: With the advancement of ancient book digitization, the number of ancient book images in databases has reached hundreds of millions. Traditional retrieval methods based on text or simple image features (such as SIFT, CNN) have high computational complexity and are difficult to achieve real-time response.

[0004] In recent years, advancements in computer vision and deep learning technologies have offered new possibilities for the identification of antique books. For example, models based on deep convolutional neural networks (CNNs) and Transformers can extract richer image features, while multimodal learning methods can combine image and text information to enhance semantic understanding. However, existing methods still face the following challenges in the application of antiquated books: Insufficient fine-grained feature modeling: The details of old books, such as damage, stains, and ink marks, are complex and varied, and existing models are unable to effectively capture these subtle differences.

[0005] Insufficient cross-modal fusion: The combination of image and text features often remains at the level of simple splicing or shallow alignment, failing to fully utilize the unique visual and semantic associations of old books.

[0006] Retrieval efficiency bottleneck: The storage and computation costs of high-dimensional feature vectors are high, and existing indexing technologies are unable to support efficient retrieval of large-scale data.

[0007] Therefore, there is an urgent need for an ancient book identification and retrieval method that combines fine-grained feature modeling, multimodal semantic fusion, and efficient indexing technology to improve retrieval accuracy and efficiency and meet the needs of cultural heritage protection and digital management. Summary of the Invention

[0008] Based on the technical problems existing in the background technology, this invention proposes an image retrieval method for ancient book covers based on dynamic segmentation and multimodal feature fusion, which realizes multi-level understanding and digital reconstruction of ancient book covers, and achieves fast retrieval through efficient index construction.

[0009] This invention proposes a method for retrieving old book cover images based on dynamic block division and multimodal feature fusion. The old book cover images are used as query vectors and input into the retrieval model to obtain a list of matching books corresponding to the old books. The training process of the retrieval model is as follows: Visual and textual features are extracted from the cover images of old books, and the images are then segmented. The segmented visual and text features are input into a cross-attention mechanism to generate an original high-dimensional fused feature vector that includes information on the layout text structure and image layout. The product quantization algorithm is used to perform subspace decomposition and quantization encoding on the original high-dimensional fused feature vector to obtain compressed feature slice vectors, which are then used to construct an ancient book feature database. Using images of old books as input query vectors, a three-level progressive query architecture is employed to index the old book database and return a list of matching books. The trainable parameters of the retrieval model are jointly optimized by constructing a total loss function.

[0010] Furthermore, the extraction of visual features from the cover images of old books specifically involves: High text density regions in the cover image are detected by a multi-scale text localization network, and the text-dominant region is obtained through multi-dimensional logical verification. Perform fine-grained chunking on the dominant text area so that each character spans at least 2×2 chunks; For large areas of damage or background areas, perform coarse-grained segmentation to reduce computational load; Each block is processed through depthwise separable convolution to extract basic features, which are then input into a channel-spatial dual attention module to obtain visual features.

[0011] Furthermore, the text-dominant region obtained through multi-dimensional logical verification is specifically as follows: Geometric layout logic: If the distance between the center points of three or more consecutive characters in a high text density area is less than 1.2 times the average font size, and the directional angle deviation of the main axis of these characters is less than 15°, then it is considered to constitute a text sequence with a linear structure, which conforms to the basic writing rules of Chinese characters in horizontal or vertical arrangement. Writing habit logic: Through the built-in lightweight layout prior knowledge module, the possible text region distribution trend of the text sequence is predicted based on the overall structure of the image, and the character group is initially located; Primary semantic clue logic: Call a fast semantic analysis submodule to analyze whether the character sequences identified in the high text density region show a valid word combination trend, so as to obtain the text dominance region after multi-dimensional logical verification.

[0012] Furthermore, the coarse-grained segmentation of large-area damage or background areas to reduce computational load specifically involves: Based on the input image of an old book cover, the following four conditions are used for determination: (1) Gradient analysis: Calculate the average Sobel gradient magnitude of the local region. If the average gradient magnitude is lower than the preset average gradient magnitude threshold, the local region is determined to be a low edge response region. (2) Texture complexity assessment: The entropy value of the texture direction distribution of the cover image is statistically analyzed using the local binary mode; or the contrast and correlation indicators are extracted by the gray-level co-occurrence matrix as texture feature indicators of each local region; if more than two-thirds of the texture feature indicators are lower than the set indicator threshold, it is judged as a low texture complexity region, which tends to be regarded as background or damaged area. (3) OCR confidence monitoring: Perform lightweight OCR recognition on the input cover image and obtain the text confidence of each local area as the basis for judgment. Record the average confidence of the recognition result. If the OCR confidence of A consecutive adjacent blocks is lower than the preset confidence threshold and there is no effective character output, it is marked as a potential non-text area. (4) Semantic segmentation-assisted verification: Input the cover image into the pre-trained semantic segmentation model to obtain the probability map of each pixel category, including the labels "stamp", "border", "white space" and "insect infestation". If a certain area is predicted to be one of the labels with a high probability, the attribute of "non-text-dominant area" is strengthened. When three or more of the four conditions are met, the local area of ​​the cover image is merged into a "non-text-dominant area" and this non-text-dominant area is treated as a large area of ​​damage or background.

[0013] Furthermore, the channel-spatial dual attention module introduces a two-dimensional sinusoidal positional encoding based on the absolute coordinates of image blocks, and adds the encoding vector element-wise to the basic features extracted by convolution to preserve the spatial relationships between blocks, specifically: The encoding is based on the absolute coordinates of each image block within the original cover image. Automatically generate encoding vectors, where Indicates row index, Indicates column index; The encoded vector is added element-wise to the base feature vector extracted by depthwise separable convolution for each block, and the resulting feature map carrying positional information is sent to the channel-space dual attention module.

[0014] Furthermore, after extracting basic features from each block through depthwise separable convolution, the data is fed into a channel-spatial dual attention module for processing to obtain visual features. Specifically: Channel attention branch: Global average pooling is performed on the input feature map in the spatial dimension to obtain a statistical summary of each channel. The dependencies between channels are learned through a fully connected network, and a set of normalized channel weight coefficients are output to reweight the response intensity of each channel. Spatial attention branch: Under the premise of keeping the channels unchanged, max pooling and average pooling are performed on the input feature map along the channel dimension to generate two two-dimensional spatial attention maps. The two attention maps are concatenated and fed into a convolutional layer to learn the importance distribution of spatial location and output spatial weight coefficients. After channel recalibrating the original input feature map using channel weight coefficients, spatial filtering is performed using spatial weight coefficients. The weighted feature map is added to the original input feature map to obtain the visual features.

[0015] Furthermore, a variable window attention mechanism is introduced into the attention calculation of the input feature maps in the channel attention branch and the spatial attention branch. The window attention mechanism is applied to the input feature maps of each layer of the Swin Transformer encoder, specifically as follows: Processing unit: Self-attention calculation is performed in units of fixed-size local windows; Sliding method: The window slides across the feature map in steps to cover the entire map, and each window independently performs multi-head self-attention operations; Cross-window communication: Introduce a "shift window" mechanism after every set number of layers; Window size growth: As the network depth increases, the original window size is expanded. The expansion criterion is: the window size doubles every two layers of network depth, forming a stable pyramid-shaped abstract structure.

[0016] Furthermore, a global semantic branch is maintained in the Swing Transformer, specifically as follows: The global semantic branch consists of a set of cross-layer connection paths. The global context features output by the high layer of the Swin Transformer are adjusted in dimension by convolution and then added or concatenated element by element with the local feature maps of the corresponding layers to obtain the local feature maps after global semantic enhancement. The local feature maps are the images corresponding to the blocks.

[0017] Furthermore, the text feature extraction of the covers of old books is performed using a multi-scale text localization network composed of feature pyramids and orientation-sensitive convolutions, specifically as follows: Feature pyramid structure: The backbone network outputs multi-level feature maps at different depths. After the multi-level feature maps are horizontally connected to a unified number of channels, they are upsampled level by level and fused with high-level features to form a five-layer pyramid structure from P2 to P6. Orientation-sensitive convolution: A set of convolutional kernels with different rotation angles are attached to each feature layer to respond to text that is tilted or arranged in an arc. Fusion method: The response maps of convolutions in each direction are stitched together and compressed through a 1×1 convolution to generate a comprehensive direction-aware feature map, which is then input into the region proposal network to generate text features.

[0018] Furthermore, the extraction of visual and textual features is used to generate an original high-dimensional fused feature vector that includes the layout text and image layout through a cross-attention mechanism, specifically as follows: A three-level cross-attention mechanism is set up, with the three levels connected in sequence. The output of the previous level serves as the input of the next level. All levels share a set of learnable projection matrices, but each has its own independent attention head configuration. By integrating the content recognition results and the visual features of the original image with the font perception reconstruction module, a gating mechanism is used to dynamically adjust the contribution weights of features from different sources during the fusion process, generating a high-dimensional fusion feature vector with standardized typesetting and font uniformity.

[0019] The advantages of this invention's method for retrieving images of ancient book covers based on dynamic segmentation and multimodal feature fusion are as follows: It employs a hybrid architecture combining dynamic segmentation and adaptive Transformer, enabling intelligent recognition of text areas and decorative patterns on ancient book covers, achieving multi-level feature extraction. A visual-language collaborative analysis mechanism is designed to address the unique variant characters, seals, and layout features of ancient books, improving recognition accuracy through dual-modal feature fusion. In the retrieval stage, an innovative multi-level index architecture is constructed, combined with an improved feature quantization method, achieving efficient and accurate retrieval of ancient book content. Furthermore, a dedicated knowledge base for ancient books is built-in, capable of automatically recognizing traditional elements such as fish-tail lines and text blocks, and possessing the ability to handle complex situations such as blurriness and tilt, providing a complete intelligent solution for the digitization of ancient books. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] like Figure 1 As shown, the present invention proposes a method for retrieving images of old book covers based on dynamic block segmentation and multimodal feature fusion, comprising: Step 1: Extract visual features and text features from the cover image of the old book and divide it into blocks. Specifically, the block division is as follows: generate fine-grained text region features and coarse-grained non-text region features based on a dynamic block division strategy. Step 2: Input the segmented visual features and text features into the cross-attention mechanism to generate the original high-dimensional fusion feature vector, which includes information on the layout text structure and image layout. Step 3: The original high-dimensional fused feature vector is decomposed into subspace and quantized using the product quantization algorithm to obtain compressed feature slice vectors, which are then used to construct an old book feature database. Step 4: Using the cover images of old books as input query vectors, index the old book database using a three-level progressive query architecture and return a list of matching books; Step 5: The trainable parameters of the retrieval model are jointly optimized by constructing a total loss function.

[0023] This embodiment employs a hybrid architecture combining dynamic segmentation and adaptive Transformer, enabling intelligent recognition of text areas and decorative patterns on ancient book covers, achieving multi-level feature extraction. For the unique variant characters, seals, and format features of ancient books, the system designs a visual-language collaborative analysis mechanism, improving recognition accuracy through dual-modal feature fusion. In the retrieval phase, the system innovatively constructs a multi-level index architecture, combined with an improved feature quantization method, achieving efficient and accurate retrieval of ancient book content. Particularly noteworthy is the system's built-in knowledge base specifically for ancient books, capable of automatically recognizing traditional elements such as fish-tail lines and text blocks, and possessing the ability to handle complex situations such as blurriness and tilt, providing a complete intelligent solution for the digitization of ancient books.

[0024] In one embodiment, step one, extracting visual and textual features from the cover image of the old book and segmenting it into blocks, specifically involves: (a1) High text density regions in the cover image are detected by a multi-scale text localization network and the dominant text region is obtained by multi-dimensional logic verification; fine-grained block division is performed on the dominant text region so that each character spans at least 2×2 blocks; Specifically, in the image feature extraction part, a multi-level processing framework of "from local to global" is adopted. First, considering the special characteristics of ancient book cover images (such as complex textures, damaged areas, seals, etc.), a content-aware dynamic segmentation mechanism is designed. This mechanism can intelligently identify key areas in the cover image. First, a multi-scale text localization network is used to initially detect high text density areas. If more than three consecutive characters within a segment are detected with a spacing between their center points less than 1.2 times the average font size, it is marked as a "high text density area". At the same time, it combines directional consistency judgment (such as character main axis angle deviation <15°) to eliminate interference from messy textures.

[0025] The system's determination of the dominant text area not only relies on visual spatial distribution, but also incorporates in-depth textual logic and layout regularity judgments to ensure the rationality of classification and semantic effectiveness.

[0026] Specifically, the identification of "high text density regions" integrates three logical verifications: geometric layout, writing habits, and primary semantic cues, and uses the results of these logical verifications as the dominant text area. Geometric layout logic: The system determines whether a text line is valid by detecting the relative distance between the center points of characters and the consistency of their arrangement direction. If the center spacing of three or more consecutive characters in a certain area is less than 1.2 times the average font size, and the angular deviation of the main axes of these characters is less than 15°, then it is considered to constitute a text sequence with a linear structure, which conforms to the basic writing rules of Chinese characters, whether horizontal or vertical.

[0027] Writing Habits and Logic: Ancient Chinese texts exhibit fixed typesetting patterns, such as "no characters at the top or bottom margins," "symmetrical layout on both sides," and "centered fish-tail" (a specific type of character). The system incorporates a lightweight prior knowledge module of layout, capable of predicting possible text area distribution trends based on the overall image structure and initially locating character groups. For example, high-density characters near the upper center of the page are more likely to be identified as the beginning of the main text paragraph; while small, dense blocks of characters in the upper right or lower left corners may be identified as title tags or bookplate annotations.

[0028] Primary semantic clue logic: After initially locating the character group, the system will call a fast semantic analysis sub-module to analyze whether the character sequence recognized within this high-text-density region shows a trend of meaningful lexical combinations (such as common ancient book terms like "rolled up", "a certain annual", "collation notes", etc.). Although it is not required to fully understand the sentence meaning, if multiple adjacent characters can form a high-frequency phrase fragment, the confidence level as an "effective text unit" is enhanced.

[0029] Therefore, the text-dominant area of this embodiment is not merely a pixel-level dense accumulation, but a text-dominant area verified through multi-dimensional logic, thereby ensuring that the object of subsequent fine-block processing is a content unit with information value.

[0030] (a2)The system dynamically adjusts the block granularity according to the region type: perform fine-grained blocking on the text-dominant area, so that each character spans at least 2×2 blocks; perform coarse-grained blocking on large damaged or background areas to reduce the computational load; For the text-dominant area, use a 7×7 fine block to capture stroke details, ensure that each character spans at least 2×2 patches, which is conducive to capturing stroke details.

[0031] It can be understood that the block refers to the basic image block (image patch) generated in the image preprocessing stage. In this embodiment, the entire cover image is first divided into basic blocks of a fixed size (such as 16×16 pixels), which serve as the basic unit for all subsequent processing.

[0032] For large damaged or background areas, use a 14×14 coarse block to improve processing efficiency. The specific process is as follows: Based on the input cover image of the ancient book, use the following four conditions for determination: (b1)Gradient analysis: Calculate the mean value of the Sobel gradient magnitude of the local area. If it is lower than the preset threshold, it indicates that there is little edge information, and it may be a blank page, a wormhole, or a severely faded area. Then, this local area is determined as a low-edge response area.

[0033] (b2)Texture complexity evaluation: Use the local binary pattern (LBP) to statistically analyze the texture direction distribution entropy value of the cover image; or extract indicators such as contrast and correlation through the gray-level co-occurrence matrix (GLCM) as the texture feature indicators of each local area; if more than two-thirds of the texture feature indicators are lower than the set index threshold, it is determined as a low-texture complexity area and is preferably regarded as the background or damaged area.

[0034] (b3) OCR Confidence Monitoring: Perform lightweight OCR parsing on the region and record the average confidence of the recognition results. If the OCR confidence of A consecutive (e.g., A=5) adjacent blocks is lower than the preset confidence threshold (e.g., 60%), and there is no valid character output, it is marked as a potential non-text region.

[0035] In this context, blocks are the basic image patches generated during the image preprocessing stage, and all basic blocks participate in the OCR confidence assessment. Therefore, this OCR confidence monitoring is applicable to any block within the entire area of ​​the cover image, aiming to identify local areas that may be unrecognizable due to severe smudges, even within areas with normal text distribution.

[0036] (b4) Semantic segmentation-assisted verification: The cover image is input into a pre-trained semantic segmentation model (based on the UNet structure, specifically for classifying elements on ancient book covers) to obtain a probability map of each pixel category, including labels such as "seal", "border", "white space", and "insect damage". If a region is predicted to be one of the above categories with a high probability, its "non-text-dominant region" attribute is strengthened.

[0037] Ultimately, the system merges regions into "non-text-dominant regions" only when three or more of the above four conditions are met. These non-text-dominant regions are then treated as large-area damage or background areas, and a 14×14 coarse-grained block strategy is employed to avoid misjudgment and significantly reduce computational overhead. Each block, after having its basic features extracted through depthwise separable convolution, is then fed into a specially designed channel-spatial dual attention module for processing.

[0038] (a3) After extracting basic features from each block through depthwise separable convolution, the blocks are input into the channel-spatial dual attention module to obtain visual features; This channel-space dual attention module is able to simultaneously focus on the importance of feature channels and the criticality of spatial location. The module adopts a parallel dual-branch structure design, focusing on modeling channel importance and spatial criticality respectively, and achieving collaborative optimization through feature fusion.

[0039] The channel-spatial dual attention module in this embodiment includes a local module and a global module, wherein the local module (residual connection) specifically comprises: Channel attention branch: First, global average pooling is performed on the input feature map in the spatial dimension to obtain a statistical summary for each channel. Then, a small-scale fully connected network (including dimensionality reduction and expansion operations) learns the dependencies between channels and outputs a set of normalized weight coefficients to reweight the response intensity of each channel. For example, if a channel mainly responds to the red ink feature of a seal, then when the network determines that the current region is the "seal area", this channel will be significantly enhanced.

[0040] Spatial Attention Branch: While keeping the number of channels constant, max pooling and average pooling are performed along the channel dimension to generate two 2D spatial attention maps. These two maps are concatenated and fed into a convolutional layer to learn the importance distribution of spatial locations. This branch can automatically focus on the most recognizable regions in the image, such as title tags, book titles, and text margins, while suppressing corner noise or background interference.

[0041] Dual-branch fusion mechanism: The output weights of the two branches are applied to the original feature map respectively—first, channel weights are applied for channel recalibration, and then spatial weights are applied for spatial filtering. The whole process forms a progressive attention control that "first considers which features are important, and then considers where the position is critical".

[0042] In this embodiment, the channel-space dual attention module also introduces a residual connection structure, which adds the original input to the weighted output, preserving the original information flow and preventing excessive suppression of useful features. This design allows the model to highlight key channels (such as those representing ink density and paper yellowing) and lock onto key positions (such as the central text area and the upper right corner title label), achieving true "channel + space" dual attention. For example, it can automatically enhance the visual features of the seal edge or title label text while suppressing background noise interference.

[0043] To maintain the spatial relationship between blocks, the channel-space dual attention module also introduces two-dimensional sinusoidal position coding. The design of two-dimensional sinusoidal position coding is derived from the classic method in the Transformer architecture, but it has been specifically optimized in this embodiment to better adapt to the fixed layout structure of ancient book covers.

[0044] The generation basis of two-dimensional sinusoidal position coding: The coding is based on the absolute coordinate position of each image patch in its original image. Automatically generated. (Among them) Indicates row index, This represents the column index. The encoding function uses a sine function for even dimensions and a cosine function for odd dimensions, with the frequency decreasing exponentially as the dimension increases, thus forming a unique positional fingerprint.

[0045] The target of the two-dimensional sinusoidal positional encoding is: the encoding vector is added element by element to the basic feature vector extracted by depthwise separable convolution of each block, that is: final input = convolutional features + positional encoding.

[0046] The purpose of introducing two-dimensional sinusoidal position encoding is to enable the model to perceive the spatial relative position of each patch in subsequent attention calculations, especially structural layouts such as "top and bottom", "left and right sides", and "fish tail in the center" commonly found in ancient books.

[0047] The relationship between 2D sinusoidal positional encoding and the channel-spatial dual attention module: Positional encoding is injected before the channel-spatial dual attention module. That is, the channel-spatial dual attention module receives an enhanced feature map already carrying positional information, enabling it to not only focus on the importance of channels but also perceive which spatial locations are more crucial (such as the central text area, the upper right corner title tag, etc.). This allows the attention mechanism not only to "see" where things are highlighted but also to "know" where those locations are. This encoding method can better preserve the unique layout information of ancient book covers.

[0048] At the global semantic understanding level, this embodiment employs an improved Swing Transformer architecture. Unlike existing conventional methods, this embodiment designs a variable window attention mechanism. The window attention mechanism applies to the input feature maps of each layer of the Swing Transformer encoder. These feature maps are derived from the output of the previous layer and are essentially a set of embedded representations of local image patches with positional encoding.

[0049] The variable window attention mechanism operates on a unit of processing: self-attention computation is performed using a fixed-size local window. For example, if the initial window is 8×8 patches, it means that within an 8×8 grid, the model will model the relationships between all patches.

[0050] The sliding method of the variable window attention mechanism: the window slides across the feature map by step size to cover the entire map, and each window independently performs multi-head self-attention operation, which greatly reduces the computational complexity of traditional global attention (from O(n²) to O(w²·n), where w is the window size).

[0051] Cross-window communication with variable window attention mechanism: In order to prevent information fragmentation, the system introduces a "shift window" mechanism after every few layers - that is, the window is shifted by half the width, so that patches that are not originally in the same window have the opportunity to interact, realizing the gradual integration of local and global information.

[0052] Therefore, the window attention mechanism is a fine-grained control over feature interactions within a local context, making it particularly suitable for capturing subtle differences in the combination of text and images on ancient book covers. The initial window size is set to 8×8 to capture fine-grained features, gradually expanding to 32×32 as the network depth increases to understand a wider range of contextual relationships. The increase in window size is not arbitrary but follows a hierarchical, progressive receptive field expansion strategy, simulating the human visual cognitive process from fine to coarse.

[0053] In shallow networks (layers 1-3), small windows (e.g., 8×8) are maintained to focus on capturing microscopic features such as stroke details, seal edges, and single character shapes. The middle-layer network (layers 4-6) gradually expands to 16×16, and begins to integrate the overall structure of entire lines of text or partial patterns; Deep networks (layers 7-9) are further expanded to 32×32, or even close to the full-image scale, to understand macro-level semantics such as page layout, theme style, and binding style.

[0054] While this expansion relationship is not expressed using explicit mathematical formulas, its design is based on empirical verification: the window size doubles every two layers of the network, forming a stable pyramid-shaped abstract structure. This strategy has been verified by numerous experiments to effectively balance computational efficiency and semantic expressiveness. More importantly, this embodiment, based on research on a large number of ancient book covers, summarizes the distribution patterns of typical elements such as fish-tail wrinkles and text margins, and encodes them into a positional bias matrix. This allows the model to automatically focus on important elements in specific locations on the cover.

[0055] Simultaneously, the system maintains a global semantic branch (i.e., a global module), which feeds back high-level semantic information to local modules through cross-layer connections, forming a closed-loop optimization of "local-global". The global semantic branch is the "top-level design hub" of the entire model, responsible for coordinating the relationship between local features and overall understanding.

[0056] The structure of the global semantic branch consists of a set of cross-layer connection paths, which originate from the global context features output by the higher layers of the SwingTransformer (usually the cls token of the last layer or the result of global pooling).

[0057] The connection method of the global semantic branch is to pass the high-level semantic features back to the local feature extraction module of the intermediate layer through skip connection. Specifically, the high-level features are adjusted in dimension by 1×1 convolution and then added or concatenated with the local feature map of the corresponding level element by element.

[0058] Information content: "High-level semantic information" refers to the model's understanding of the entire cover, such as abstract concepts like "this is a Qing Dynasty woodblock print", "the cover has three red seals", and "belongs to the style of Jinling Bookstore". The mechanism of global semantic branches: This information serves as "prior guidance," feeding back to local modules and making them more directional when extracting details. For example, once the model realizes that the overall style leans towards Ming Dynasty blue-printed texts, it will automatically increase its attention to light-inked areas, improving the sensitivity of recognizing blurred text.

[0059] This forms a closed-loop optimization architecture of "bottom-up perception → top-down regulation", which significantly enhances the system's context awareness capability.

[0060] (a4) Extract text features from the cover images of old books; In terms of text processing, the system faces unique challenges specific to ancient texts: variant characters, slanted arrangement, and ink smudges. To address these challenges, a multi-scale text localization network was developed. This network integrates a feature pyramid structure and orientation-sensitive convolution. The network adopts a dual-track fusion design of "feature pyramid + orientation-sensitive convolution" to balance scale diversity and orientation adaptability.

[0061] Feature pyramid structure: The backbone network (such as ResNet-50) outputs multi-level feature maps (C2-C5) at different depths, with resolution decreasing sequentially and semantics gradually increasing. These feature maps are then upsampled level by level through lateral connections to unify the number of channels and fused with higher-level features to form a five-layer pyramid structure (P2-P6), capable of detecting text of various sizes, from small annotations to large titles.

[0062] Orientation-sensitive convolution: A set of convolutional kernels with different rotation angles (such as 0°, 30°, 45°, 60°, 90°) are attached to each feature layer. These convolutional kernels are specifically designed to respond to text that is tilted or arranged in an arc (common in the marginal notes and side notes of ancient books). Fusion method: The response maps of convolutions in each direction are stitched together, and then the channels are compressed through 1×1 convolution to finally generate a comprehensive direction-aware feature map, which is then input into RPN (Region Proposal Network) to generate text features.

[0063] This multi-scale text localization network enables this embodiment to automatically adapt to complex layouts without prior knowledge of the text direction, greatly improving detection robustness and allowing for adaptive detection of text regions of different sizes and directions.

[0064] In the recognition phase, which specifically refers to the actual inference process after the model has been trained and deployed, an image of an old book cover to be retrieved is input. The system sequentially executes: image preprocessing → dynamic segmentation → local feature extraction → text localization and recognition → multimodal feature fusion → feature vectorization. All modules run with frozen parameters and do not update weights. The output is a standardized multimodal feature vector for subsequent indexing queries. This phase emphasizes real-time performance and stability; all calculations are forward propagation and do not involve backward gradient updates, completely different from the data iteration optimization process in the training phase. Therefore, the innovative visual-language dual-stream architecture plays a crucial role: the visual branch focuses on extracting glyph features, accurately locating each character through character-level masks; the language branch utilizes a pre-trained ancient book language model to provide semantic priors, and the two branches interact dynamically through an attention mechanism. This design significantly improves the recognition accuracy of rare characters and ambiguous text.

[0065] In one embodiment, step two involves inputting the extracted visual and textual features into a cross-attention mechanism to generate an original high-dimensional fusion feature vector that includes information on the page layout and text structure; specifically: To achieve deep integration of visual and textual features, the system constructs a three-level cross-attention mechanism. At the micro level, character mask constraints are introduced to force attention to the region where the current character is located, preventing misassociations with neighboring text and achieving accurate matching between individual characters and local image features. At the meso level, a bidirectional attention flow is employed, where the image guides text decoding, and the text, in turn, corrects the image region weights, forming a mutually reinforcing relationship between text lines and their corresponding visual regions. At the macro level, a category-prior gating is added to adjust the visual-semantic alignment strength according to the book type (history, classics, philosophy, literature) to ensure the consistency between the visual presentation of the entire document and the theme of the text content. Structurally, the three attention layers are sequentially connected, with the output of the previous layer serving as the input for the next. For example, the fine alignment results from the micro layer are injected into the query vector of the meso layer, forming a "details-driven overall" linkage effect.

[0066] In addition, all levels share a set of learnable projection matrices, but each has its own independent attention head configuration, ensuring a balance between specialization and generalization.

[0067] Meanwhile, the font-aware reconstruction module can integrate content recognition results and original image style features to generate standard text images with standardized layout and uniform fonts, which facilitates subsequent digital storage and display. The innovation of this module is mainly reflected in two aspects: structural design and functional integration.

[0068] Regarding the structure of the font-aware reconstruction module: The module adopts a dual encoder-single decoder architecture: the image encoder extracts the style features of the original glyphs, such as texture, thickness, and ink density; the text encoder obtains the content information of standard Unicode characters; and the decoder merges the two to generate a reconstructed image that combines "correct content" and "original style".

[0069] Introducing a style transfer bottleneck layer: A low-dimensional latent space is inserted during the fusion process, forcing the model to extract the most essential font style factors (such as the skeletal feel of Song typeface and the roundness of Kai typeface), thus achieving controllable style generation.

[0070] Functions of the font-aware reconstruction module: It can not only repair blurry and broken text, but also unify the layout style, transforming messy handwritten text into neat printed text; it supports style transfer options, allowing users to choose "restor original appearance" or "standardized output"; the output can be used in various scenarios such as digital exhibitions and publishing reproduction, filling the gap of traditional OCR that only outputs text strings.

[0071] Therefore, this font-aware reconstruction module is not only a recognition tool but also a core component for the digital reconstruction of cultural heritage. In the feature fusion and optimization stage, a gating mechanism is used to dynamically adjust the contribution weights of features from different sources. The gating mechanism is a context-aware adaptive weighting system that does not require explicit formulas and relies on neural networks to autonomously learn control strategies.

[0072] The gating mechanism is implemented as follows: Construct a small gating network (usually several fully connected layers), with inputs including: local visual features of the current block; corresponding text recognition confidence; region type label (from semantic segmentation); surrounding environmental context (such as whether there are stamps, borders, etc. in the vicinity); The gated network outputs two normalized weight coefficients ( These correspond to the contribution ratios of "local detail features" and "global context features," respectively. Final fusion feature = ×Local features+ ×Global features.

[0073] During training, the model automatically learns to tune parameters in different contexts using supervisory signals: in text regions, Significantly greater than Emphasizing the clarity of strokes; in the decorative pattern area, The dominant feature structure emphasizes the overall artistic style; however, at the edges of damage, a balance is struck, considering both content restoration and visual harmony. The entire process is data-driven, requiring no manual rule setting, demonstrating the powerful adaptability of deep learning. For example, the weight of local detail features is enhanced in text areas, while global contextual information is relied upon more heavily in decorative pattern areas. The bidirectional feature pyramid structure ensures full interaction of features at different scales, enabling the system to recognize both fine seal characters and the overall layout.

[0074] In one embodiment, step three involves using a product quantization algorithm to perform subspace decomposition and quantization encoding on the original high-dimensional fused feature vector to obtain a compressed feature slice vector, which is then used to construct an old book feature database. Specifically: To achieve efficient indexing, an improved product quantization algorithm is first used to process the original high-dimensional fused feature vector. Data compression is achieved through two key steps: feature space decomposition and subspace quantization. Specifically, the algorithm uniformly divides the input feature vector into multiple subvectors, and an optimized codebook is trained for each subspace. This system has made three key improvements to the traditional subspace codebook training of product quantization.

[0075] (c1) Non-uniform subspace partitioning: Traditional methods divide feature vectors into several segments; this embodiment, based on the variance distribution of each dimension, groups high-variance dimensions into the same subspace, thereby improving quantization accuracy.

[0076] (c2) Weighted K-means clustering: When clustering within each subspace, different weights are assigned to different samples; frequently occurring features (such as common layouts) receive higher weights to ensure better codeword matching for mainstream patterns.

[0077] (c3) Incremental codebook update: The codebook supports online fine-tuning. When a new batch of ancient books is added, the codebook can be quickly updated through a small number of iterations without retraining the entire indexing system.

[0078] These improvements significantly enhanced the quantified retrieval recall, particularly when dealing with less popular versions. This divide-and-conquer strategy drastically reduced storage requirements while preserving key features.

[0079] In one embodiment, step four involves using the cover images of old books as the input query vector, indexing the old book database using a three-level progressive query architecture, and returning a list of matching books. Specifically: A three-level progressive query architecture was designed: the first level is based on the inverted index to obtain the candidate set; the second level is based on the coarse ranking of similarity using Locality Sensitive Hash (LSH); and the third level is based on the fine ranking of Approximate Nearest Neighbor Search (ANN) using a graph structure.

[0080] The coarse-grained filtering process of the first-level inverted index is as follows: Index building phase: The feature vectors of all books are quantized and transformed into a "bag of words" format, with the most recent codeword in each subspace being a "word"; a mapping table (i.e., an inverted index) is established for "word → book ID list". Filtering logic during the query phase: The features of the query image are also quantified in the same way to obtain a set of "query terms"; a set of candidate books corresponding to each "query term" is retrieved; the union of these sets is taken to form an initial candidate set; then, the books are sorted according to their co-occurrence frequency, and books that match the most query terms are retained first; a threshold is set to remove entries with a co-occurrence frequency below a certain level (such as books that match only one word), thus completing coarse-grained filtering. This approach can quickly reduce a database of hundreds of millions to thousands, laying the foundation for subsequent fine screening.

[0081] This embodiment uses the classic random projection type LSH to achieve the second level of precise matching: multiple hash tables are constructed, and each table uses a set of random hyperplanes to partition the feature space; the probability that the query vector and the database vector fall in the same bucket is positively correlated with their similarity; in the candidate set after inverted filtering, only the precise distance (such as cosine similarity) between those entries that are in the same hash bucket as the query vector is calculated; the Top-K results with the highest scores are returned as the final matching list.

[0082] This method can significantly reduce the number of distance calculations while ensuring a high recall rate, achieving millisecond-level response.

[0083] Intelligent connections between indexes at each level are achieved through a dynamic routing mechanism: the first level (inverted index) quickly returns approximately 100,000 candidates; the controller analyzes the rarity of the query vector (judged by its distance distribution from the codebook): if it is a common pattern, the inverted index is skipped directly, and LSH fine filtering is initiated; if it is a rare or ambiguous image, the second and third levels are activated for secondary filtering; simultaneously, the recall quantity at each level is dynamically adjusted according to the current server load to avoid resource overload; the entire process is like a "funnel + diversion valve," ensuring both high efficiency and high accuracy. The system automatically adjusts the search path and computational resource allocation based on the characteristics of the query vector. The system has a built-in query complexity evaluation module, which is essentially a lightweight classifier used to predict query difficulty. Input to the query complexity evaluation module: basic statistical characteristics of the query vector (such as energy distribution, sparsity, and initial distance to various typical templates); The query complexity assessment module outputs three levels: "easy", "moderate", and "hard".

[0084] Adjustment strategy for the query complexity evaluation module: "Simple" query: Enables high-speed path, skips unnecessary levels, and matches directly; "Difficult" query: Enable full-path depth search, increase the candidate set size, and enable high-precision matching algorithm; Parallel scheduling: For "general" queries, attempt multi-path parallel computation and return the best result.

[0085] This adaptive mechanism enables the system to achieve the best balance between speed and accuracy, meeting the needs of diverse practical application scenarios.

[0086] In one embodiment, step five, retrieving the trainable parameters of the model, is jointly optimized by constructing a total loss function, specifically as follows: A total loss function is constructed, which includes visual feature reconstruction loss, text alignment loss, and multimodal consistency loss, to jointly optimize the trainable parameters of the retrieval model.

[0087] In a preferred embodiment, the total loss function constructed in this embodiment consists of visual reconstruction loss. Text alignment loss and multimodal consistency loss The three parts are weighted and combined, and their overall form can be expressed as: ; in: The weighting coefficient is a hyperparameter that can be set within a reasonable range (e.g., 0.3–1.0) based on experimental experience. It is used to adjust the contribution ratio of the three types of loss in the overall optimization process, so as to achieve joint training and dynamic balance between branches.

[0088] (1) Visual feature reconstruction loss (Corresponding to the visual feature extraction optimization in step one); This is used to constrain the feature reconstruction quality of image branches. This embodiment uses mean squared error (MSE) to evaluate the original visual feature vector. With reconstructing feature vectors Measure the differences: ; Where N is the feature dimension, The first The original visual feature vector and the reconstructed feature vector are calculated. This loss is used to preserve the layout structure, font texture, and ancient book cover layout information carried by the visual features, thereby enhancing the feature representation ability of the visual encoding network.

[0089] (2) Text alignment loss (Corresponding to step two: optimization of the cross-attention module); This is used to constrain the alignment relationship between visual features and text features in a unified semantic space. This embodiment employs the InfoNCE contrastive learning loss to constrain the alignment relationship between visual features of the query image. Corresponding text features Improve similarity: ; in, For similarity function, For temperature coefficient, These are negative sample text features.

[0090] This loss improves the similarity of positive samples and suppresses the similarity of negative samples, thereby achieving cross-modal semantic alignment between visual and textual features and optimizing the fusion capability of the cross-attention mechanism.

[0091] (3) Multimodal consistency loss (Corresponding to multimodal feature fusion) To maintain consistency in the prediction distribution between the visual and text branches, this embodiment uses the KL divergence form: ; in and Representing visual branches and text branches for categories respectively. The predicted probability is calculated. This loss is used to constrain the consistency of probabilities between the two modalities in the semantic space, thereby improving the stability and fusion accuracy of the final multimodal features.

[0092] (4) Technical improvements of the loss function in this embodiment; Compared to existing technologies that generally employ single reconstruction loss or cross-entropy loss, this embodiment jointly models three types of losses: visual reconstruction, semantic alignment, and cross-modal consistency. These losses are then applied to the visual encoding network, the cross-attention module, and the multimodal fusion module, respectively, thereby achieving synergistic optimization of layout structure, character semantics, and cross-modal consistency. This combined loss function significantly improves the robustness and accuracy of retrieval of old book covers in scenarios with complex text, mixed text and images, and damaged backgrounds.

[0093] The ancient book image retrieval method proposed in this embodiment has the following innovative advantages: It adopts a hybrid architecture combining dynamic segmentation and adaptive Transformer, enabling intelligent recognition of text areas and decorative patterns on ancient book covers, achieving multi-level feature extraction. For the unique variant characters, seals, and layout features of ancient books, this embodiment designs a visual-language collaborative analysis mechanism, improving recognition accuracy through dual-modal feature fusion. In the retrieval stage, this embodiment innovatively constructs a multi-level index architecture, combined with an improved feature quantization method, achieving efficient and accurate retrieval of ancient book content. It is particularly noteworthy that the system has a built-in knowledge base specifically for ancient books, capable of automatically recognizing traditional elements such as fish-tail lines and text blocks, and possessing the ability to handle complex situations such as blurriness and tilt, providing a complete intelligent solution for the digitization of ancient books.

[0094] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for retrieving images of old book covers based on dynamic block segmentation and multimodal feature fusion, characterized in that, The images of old book covers are used as query vectors and input into the retrieval model to obtain a list of matching books corresponding to the old books. The training process of the retrieval model is as follows: Visual and textual features are extracted from the cover images of old books, and the images are then segmented. The segmented visual and text features are input into a cross-attention mechanism to generate an original high-dimensional fused feature vector that includes information on the layout text structure and image layout. The product quantization algorithm is used to perform subspace decomposition and quantization encoding on the original high-dimensional fused feature vector to obtain compressed feature slice vectors, which are then used to construct an ancient book feature database. Using images of old books as input query vectors, a three-level progressive query architecture is employed to index the old book database and return a list of matching books. The trainable parameters of the retrieval model are jointly optimized by constructing a total loss function.

2. The method for retrieving images of ancient books according to claim 1, characterized in that, The extraction of visual features from the cover images of old books specifically involves: High text density regions in the cover image are detected by a multi-scale text localization network, and the text-dominant region is obtained through multi-dimensional logical verification. Perform fine-grained chunking on the dominant text area so that each character spans at least 2×2 chunks; For large areas of damage or background areas, perform coarse-grained segmentation to reduce computational load; Each block is processed through depthwise separable convolution to extract basic features, which are then input into a channel-spatial dual attention module to obtain visual features.

3. The method for retrieving images of ancient books according to claim 1, characterized in that, The text-dominant region obtained through multi-dimensional logical verification is specifically as follows: Geometric layout logic: If the distance between the center points of three or more consecutive characters in a high text density area is less than 1.2 times the average font size, and the directional angle deviation of the main axis of these characters is less than 15°, then it is considered to constitute a text sequence with a linear structure, which conforms to the basic writing rules of Chinese characters in horizontal or vertical arrangement. Writing habit logic: Through the built-in lightweight layout prior knowledge module, the possible text region distribution trend of the text sequence is predicted based on the overall structure of the image, and the character group is initially located; Primary semantic clue logic: Call a fast semantic analysis submodule to analyze whether the character sequences identified in the high text density region show a valid word combination trend, so as to obtain the text dominance region after multi-dimensional logical verification.

4. The method for retrieving images of ancient books according to claim 2, characterized in that, The method of performing coarse-grained segmentation on large-area damaged or background areas to reduce computational load is as follows: Based on the input image of an old book cover, the following four conditions are used for determination: (1) Gradient analysis: Calculate the average Sobel gradient magnitude of the local region. If the average gradient magnitude is lower than the preset average gradient magnitude threshold, the local region is determined to be a low edge response region. (2) Texture complexity assessment: The entropy value of the texture direction distribution of the cover image is statistically analyzed using the local binary mode; or the contrast and correlation indicators are extracted by the gray-level co-occurrence matrix as texture feature indicators of each local region; if more than two-thirds of the texture feature indicators are lower than the set indicator threshold, it is judged as a low texture complexity region, which tends to be regarded as background or damaged area. (3) OCR confidence monitoring: Perform lightweight OCR recognition on the input cover image and obtain the text confidence of each local area as the basis for judgment. Record the average confidence of the recognition result. If the OCR confidence of A consecutive adjacent blocks is lower than the preset confidence threshold and there is no effective character output, it is marked as a potential non-text area. (4) Semantic segmentation-assisted verification: Input the cover image into the pre-trained semantic segmentation model to obtain the probability map of each pixel category, including the labels "stamp", "border", "white space" and "insect-eaten". If a certain area is predicted to be one of the labels with a high probability, the attribute of "non-text-dominant area" is strengthened. When three or more of the four conditions are met, the local area of ​​the cover image is merged into a "non-text-dominant area" and this non-text-dominant area is treated as a large area of ​​damage or background.

5. The method for retrieving images of old books according to claim 2, characterized in that, The channel-spatial dual attention module introduces a two-dimensional sinusoidal positional encoding based on the absolute coordinates of image blocks, and adds the encoding vector element-wise to the basic features extracted by convolution to preserve the spatial relationships between blocks. Specifically: The encoding is based on the absolute coordinates of each image block within the original cover image. Automatically generate encoding vectors, where Indicates row index, Indicates column index; The encoded vector is added element-wise to the base feature vector extracted by depthwise separable convolution for each block, and the resulting feature map carrying positional information is sent to the channel-space dual attention module.

6. The method for retrieving images of ancient books according to claim 2, characterized in that, The process involves extracting basic features from each block using depthwise separable convolution and then feeding them into a channel-spatial dual attention module for processing to obtain visual features. Specifically: Channel attention branch: Global average pooling is performed on the input feature map in the spatial dimension to obtain a statistical summary of each channel. The dependencies between channels are learned through a fully connected network, and a set of normalized channel weight coefficients are output to reweight the response intensity of each channel. Spatial attention branch: Under the premise of keeping the channels unchanged, max pooling and average pooling are performed on the input feature map along the channel dimension to generate two two-dimensional spatial attention maps. The two attention maps are concatenated and fed into a convolutional layer to learn the importance distribution of spatial location and output spatial weight coefficients. After channel recalibrating the original input feature map using channel weight coefficients, spatial filtering is performed using spatial weight coefficients. The weighted feature map is added to the original input feature map to obtain the visual features.

7. The method for retrieving images of ancient books according to claim 6, characterized in that, In the attention calculation of the input feature maps by the channel attention branch and the spatial attention branch, a variable window attention mechanism is introduced. The window attention mechanism is applied to the input feature maps of each layer of the Swing Transformer encoder, specifically as follows: Processing unit: Self-attention calculation is performed in units of fixed-size local windows; Sliding method: The window slides across the feature map in steps to cover the entire map, and each window independently performs multi-head self-attention operations; Cross-window communication: Introduce a "shift window" mechanism after every set number of layers; Window size growth: As the network depth increases, the original window size is expanded. The expansion criterion is: the window size doubles every two layers of network depth, forming a stable pyramid-shaped abstract structure.

8. The method for retrieving images of ancient books according to claim 7, characterized in that, In the Swing Transformer, a global semantic branch is maintained, specifically as follows: The global semantic branch consists of a set of cross-layer connection paths. The global context features output by the high layer of the Swin Transformer are adjusted in dimension by convolution and then added or concatenated element by element with the local feature maps of the corresponding layers to obtain the local feature maps after global semantic enhancement. The local feature maps are the images corresponding to the blocks.

9. The method for retrieving images of ancient books according to claim 1, characterized in that, The text feature extraction of the covers of old books is performed using a multi-scale text localization network composed of feature pyramids and orientation-sensitive convolutions. Specifically: Feature pyramid structure: The backbone network outputs multi-level feature maps at different depths. After the multi-level feature maps are horizontally connected to a unified number of channels, they are upsampled level by level and fused with high-level features to form a five-layer pyramid structure from P2 to P6. Orientation-sensitive convolution: A set of convolutional kernels with different rotation angles are attached to each feature layer to respond to text that is tilted or arranged in an arc. Fusion method: The response maps of convolutions in each direction are stitched together and compressed through a 1×1 convolution to generate a comprehensive direction-aware feature map, which is then input into the region proposal network to generate text features.

10. The method for retrieving images of ancient books according to claim 1, characterized in that, The process of generating an original high-dimensional fused feature vector, including layout text and image composition, from the extracted visual and textual features through a cross-attention mechanism is as follows: A three-level cross-attention mechanism is set up, with the three levels connected in sequence. The output of the previous level serves as the input of the next level. All levels share a set of learnable projection matrices, but each has its own independent attention head configuration. By integrating the content recognition results and the visual features of the original image with the font perception reconstruction module, a gating mechanism is used to dynamically adjust the contribution weights of features from different sources during the fusion process, generating a high-dimensional fusion feature vector with standardized typesetting and font uniformity.

Images