Intelligent old newspaper discovery method and system based on multi-modal clue probability fusion

Abstract

The invention discloses an intelligent old newspaper discovery method and system based on multi-modal clue probability fusion, and relates to a digital library and information retrieval technology. In order to solve the retrieval problem caused by low OCR (Optical Character Recognition) quality of old newspapers, a scheme of'system noise tolerance, probability reference and man-machine collaboration 'is adopted, text, vision and structure multi-modal clues are extracted through a data preprocessing subsystem, an index storage subsystem constructs a mixed architecture index, a query processing and sorting subsystem calculates correlation on the basis of a probability fusion model, and a retrieval result is obtained. The man-machine collaborative feedback subsystem dynamically optimizes the weight, and high availability is guaranteed through containerized deployment. According to the method, OCR errors do not need to be corrected, through multi-clue probability fusion and user feedback optimization, high-recall-rate retrieval in a low-quality OCR environment is achieved, multi-mode discovery is supported, and the utilization efficiency of old newspapers and literatures is improved.

Description

Technical Field

[0001] This invention relates to digital library and information retrieval technologies, specifically a method and system for retrieving and ranking old newspaper documents with low optical character recognition quality by integrating multimodal clues and introducing a human-computer collaboration mechanism to improve discovery efficiency and user experience. Its core lies in tolerating OCR noise through technical means, providing users with probabilistic decision-making references, and ultimately relying on the user's professional judgment to complete information discovery. Background Technology

[0002] Since the 1980s, my country's public libraries have launched a microfilm preservation project for old newspapers, accumulating tens of millions of high-resolution images to date, laying the foundation for the long-term preservation of these documents. However, the "searchability" of these images is far lower than their "preservability," with the fundamental bottleneck lying in the severely lagging development of the "content layer."

[0003] 1. The combination of outdated carriers and early digitization conditions has led to widespread problems in images, such as bleed-through, damage, yellowing, and low contrast. Traditional OCR engines have an average character recognition rate of less than 75% on such scanned documents, and the error rate varies significantly by region—the error rate in key areas such as titles, advertisements, and image descriptions is higher than that in the main text, directly causing a structural inversion of "high-value information represented by low quality".

[0004] 2. To improve the utilization rate of old newspapers, the traditional practice of libraries is to organize manual reformatting and data entry of the text content. This method is costly and inefficient. More importantly, manual data entry can only cover plain text information. It cannot effectively identify and retrieve non-textual elements in newspapers that carry important historical value, such as advertisements, news photos, cartoons, and seals. As a result, this high-value information has long been in a state of "undiscoverable".

[0005] 3. The traditional “full-text index + keyword retrieval” model treats OCR output as “near-gold” data, without explicitly modeling the error. The recall rate decreases exponentially with the error rate. Users have to give up after multiple unsuccessful searches, and the digital investment is difficult to be transformed into research output.

[0006] 4. Although the multimodal RAG solution introduces visual encoding, it still directly embeds the OCR text into the vector space and adopts a "hard ignore" strategy for the remaining 25% of typos. The error ceiling still exists, and it does not solve the problem of how non-textual cues such as "images, tables, stamps, comics, and musical scores" can work together with textual cues.

[0007] 5. With the maturity of end-to-end visual language models, Tencent's open-source Hunyuan OCR model (HunyuanOCR) in 2025 improved the general scene recognition rate to 90% with 1B parameters, which can significantly reduce noise at the source. However, the model only outputs the "best text" and does not provide character-region credibility, semantic labels of layout elements (photos, cartoons, seals, dividing lines) and cross-modal relevance measurement. Therefore, even if the OCR characters are correct, the system still cannot answer in-depth search needs such as "Does this cartoon satirize the same event?" or "Is this portrait photo related to the keyword person?"

[0008] 6. Existing patents (such as CN118247798A) only perform topic relevance statistics on OCR elements, without establishing a complete technical link of "fragmented clues - probability fusion - accurate sorting" for user query scenarios, and without introducing a human-machine collaboration mechanism to perform closed-loop optimization for residual uncertainties.

[0009] In summary, the digitization of old newspapers has completed the "image preservation" stage and urgently needs a new technical system that is "highly noise-tolerant, probabilistic, and human-machine collaborative." This system should integrate irreparable OCR errors, visual semantic gaps, and user needs into a probabilistic framework to achieve high recall and high reliability in low-quality data environments. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent discovery scheme for old newspaper documents with high robustness to OCR noise, aiming to achieve high recall, support multiple discovery modes, and ensure system performance and availability.

[0011] This invention establishes a design concept, algorithm, and solution based on "system noise tolerance, probabilistic reference, and human-machine collaboration":

[0012] 1. System noise tolerance: Model it directly using a probabilistic model (including OCR errors), rather than attempting to completely correct it;

[0013] 2. Probability Reference: Preliminary search results are presented in the form of newspaper metadata and relevance probability, providing users with a reference for decision-making strength;

[0014] 3. Human-computer collaboration: The system optimizes the sorting by displaying original newspaper images and responding to user feedback through online learning. At the same time, it gives the final information confirmation right to the user, thus realizing a solution in a low-quality OCR environment that replaces exact matching with probabilistic retrieval and replaces the system's one-way output with human-computer collaborative decision-making.

[0015] (I) System Architecture

[0016] A multimodal probabilistic fusion intelligent discovery system for low-quality OCR old newspaper documents (overall architecture see...) Figure 1(as shown), characterized in that it includes:

[0017] 1. Data preprocessing subsystem, including:

[0018] - Image Quality Assessment Module: Used to grade the quality of scanned document images, outputting a comprehensive quality score Q and grade labels (e.g., Excellent, Good, Average, Poor). An end-to-end quality scoring pre-training is performed using a CNN model (e.g., ResNet), with evaluation metrics including blurriness, contrast, noise level, and degree of smudges.

[0019] - Page Layout Analysis Module: Based on deep learning models (YOLOv8, Faster R-CNN, or DETR, fine-tuned on old newspaper page layout datasets), it identifies and locates article areas, title areas, image areas, and advertising areas in documents.

[0020] - Multimodal cue extraction module: used to extract textual cues (including titles, dates, and named entities), visual cues (2048-dimensional depth feature vectors extracted from image regions using ResNet or CLIP-ViT models), and structural cues (page layout and region coordinates (x1, y1, x2, y2)) in parallel; among which, textual cues are extracted using OCR tools, dates are extracted using regular expressions or NER models, and named entities are identified using NER models finely tuned on historical text.

[0021] 2. The index storage subsystem is used to build a multi-clue inverted index. This index stores the content of each clue, the coordinates of the relevant document and region, and the initial confidence score of that clue. It adopts a hybrid architecture of Elasticsearch (supporting complex indexes and efficient retrieval) and Milvus (specifically for vector indexing), using "document fragments" (such as an article and its accompanying figures) as the basic index unit. Index fields include doc_id (unique document identifier), text_title (title text, word segmentation), text_content (full text, word segmentation), entity_* (named entities), publish_date (publication date), visual_feature (image region vector features), layout_region (region type and coordinates), confidence_score (initial confidence score of the clue), and quality_score (image quality score Q).

[0022] 3. Query processing and sorting subsystem, including:

[0023] - Multimodal Query Understanding Module: Used to parse user input of natural language or images, generating structured query cue vectors. Text queries use NLU (Non-Limited Algorithm) technology to parse keywords, dates, locations, and query intent; image queries use the same model as visual cue extraction to extract deep feature vectors.

[0024] - Probabilistic fusion ranking engine: This is the core module. Based on the method described in claim 2, it calculates the relevance probability between document fragments and queries. It ensures that 95% of query requests return results within 500ms and supports an index throughput of thousands of pages of documents per hour.

[0025] 4. Human-Machine Collaborative Feedback Subsystem: This subsystem collects user clicks, dwell time, and confirmation actions related to search results. Based on these feedback signals, it dynamically updates the cue weight matrix in the probabilistic fusion ranking engine using an online learning algorithm. The weights are updated using stochastic gradient descent, and a learning rate decay and weight change monitoring mechanism is implemented to prevent noise feedback and concept drift. Weight updates are transmitted in real-time via POST / api / v1 / feedback interfaces, and the triggering mechanism supports periodic or real-time fine-tuning.

[0026] 5. Deployment and Operation Module: Docker containerization is adopted, and Kubernetes is used for service orchestration. It supports horizontal scaling of indexing and retrieval services to cope with data growth. Prometheus and Grafana are integrated to monitor QPS, latency, and error rate. ELK Stack is used to collect and analyze logs to ensure that the overall system availability is not less than 99.9%.

[0027] (II) Index Building Process

[0028] The complete construction of an index consists of four stages: clue preprocessing and normalization, initial confidence calculation, hybrid architecture index creation, and incremental updates, as detailed below:

[0029] 1. Clue Preprocessing and Standardization

[0030] - Text clue standardization: The title and full text extracted by OCR are segmented into words (the word list is optimized to adapt to the Chinese context), and stop words are filtered. Named entities and dates are normalized to ensure search consistency.

[0031] - Visual cue standardization: The 2048-dimensional visual feature vector is L2 normalized (ensuring the vector magnitude is uniformly 1, improving the efficiency of cosine distance calculation); at the same time, semantic labels (such as "advertisement image", "portrait", "cartoon", "news photo", "stamp") are added to the image region. The specific method is as follows: based on the labeled old newspaper image dataset (including the above semantic labels for classification samples), a lightweight classification model (such as MobileNetV3, EfficientNet-BO) is used for supervised training. During the training process, the cross-entropy loss function is used to optimize the model parameters. The input is the 2048-dimensional visual feature vector of the image region (extracted using the same model as the cue extraction stage), and the output is the semantic label and label confidence (value range [0, 1]) corresponding to the image region; only semantic labels with confidence ≥ 0.6 are stored in the index, and those below this threshold are marked as "unclassified image", which improves the matching accuracy and label reliability.

[0032] - Structural cue standardization: Converts regional coordinates to relative coordinates to avoid retrieval bias caused by differences in image resolution.

[0033] 2. Initial confidence score s core) calculation

[0034] The initial confidence level for a clue is calculated based on a multi-dimensional factor weighting, using the following formula:

[0035] confidence s core=α×BaselineAccuracy(r)×(1-HistoricalErrorRate(o i ))+β×Quality S core(Q);

[0036] Where α = 0.7 and β = 0.3 are weighting coefficients; OCR A ccuracy(r) is the prior recognition accuracy of region r; HistoricalErrorRate(o i () represents the historical error rate of the clues; Quality S core(Q) is the image quality normalization score.

[0037] 3. Hybrid Schema Index Creation

[0038] - Elasticsearch indexes: store text, structural clues, and metadata, and support fuzzy matching, exact matching, and range queries.

[0039] -Milvus Vector Index: Stores visual feature vectors and builds an IVF_FLAT index to ensure efficient similarity queries.

[0040] - Index verification: Sample 10% of the data to verify the accuracy of associations and ensure that the recall rate is not less than 90%.

[0041] 4. Incremental index update mechanism

[0042] Incremental updates are triggered when new data or model parameters are added. The index is updated through the process of "preprocessing → clue extraction → standardization → confidence calculation → incremental merging". A read-write separation strategy is adopted to ensure service continuity, and the update time is ≤30 minutes / 1000 pages.

[0043] (III) Core Retrieval Algorithm

[0044] A multimodal cue probability fusion ranking method for the system of claim 1, characterized by comprising the following steps (see...) Figure 2 As shown):

[0045] Step S201: Clue confidence modeling for g i An arbitrary cue observation o extracted from a literature region r with quality level Q. i The conditional probability reflecting true information follows the model below:

[0046] P(o i |g i ,Q,r)=exp(-λ i (Q, r)*d(o i g i )) / Z

[0047] Where d(·,·) is the metric function, using normalized edit distance for textual cues and cosine distance for visual cues; Z is the normalization constant; λ i The precision control factor is defined as follows:

[0048] λ i (Q, r)=Q*BaselineAccuracy(r) / (1+HistoricalErrorRate(o i ))

[0049] Where BaselineAccuracy(r) is the prior recognition accuracy of region r, and HistoricalErrorRate(o i The error rate is the historical statistical rate for this type of clue.

[0050] Step S202: Calculate the relevance probability of a single clue given a query clue q i With literature clues i Its non-exact matching kernel function is defined as: K(q) i o i)=exp(-γ*d(q i o i ) p Where p is a smoothing factor, with a value range of [0.5, 0.7]. Therefore, the relevance probability of a single clue is determined by both the matching degree and the confidence level: P i (rel|q i o i )=K(q i o i )*P(o i |g i ,Q,r)

[0051] Step S203: The multi-clue probability fusion model assumes that each clue is conditionally independent under given relevance conditions. Therefore, the fused relevance probability adopts a weighted Logit model: P(rel|q, o)=σ(∑(w i *logit(P i (rel|q i o i )))+b) where σ(·) is the Sigmoid function, logit(x)=log(x / (1-x)), w i Let b be the dynamic weight of clue i, and b be the global bias term.

[0052] Step S204: The weight w described in the online weight learning mechanism i Optimization is achieved through online learning. The loss function is defined as cross-entropy loss: L = -[y*log(P) + (1-y)*log(1-P)] where y∈0,1 represents the true user feedback labels (e.g., 1 for clicks and dwell time, 0 for exposures without clicks or quick skips). Based on stochastic gradient descent, the weight update formula is: w i ←w i -η*(Py)*logit(P i Where η is the learning rate. This system achieves adaptive evolution of the weight matrix by continuously collecting user feedback. (iv) Implementation Examples

[0053] For example, a user might search for "advertisements for betel nuts and Vienna-made cigarettes published in Guangzhou newspapers in 1935":

[0054] 1. The query comprehension module parses out text clues {q t = "Betel Nut Vienna Cigarettes"}、q d =“1935”, {q e The query intent was identified as including images, specifically "Guangzhou". NLU (Non-Local Logic) technology was used to perform intent recognition and named entity extraction, generating a structured query clue vector q = q t qd q e Image intent.

[0055] 2. In the index, the clue for a certain document fragment s is: title o t = "Betel Nut Vienna Cigarettes", Date o d = "1935", Location o e = "Guangzhou", and contains an image region whose 2048-dimensional visual features v extracted by ResNet o Highly correlated with "advertising"; the image quality score of this document fragment is Q = 0.7, the baseline accuracy of title region prior recognition is BaselineAccuracy = 0.85, and the historical error rate of title clues is HistoricalErrorRate = 0.1.

[0056] 3. The probability fusion sorting engine performs the calculations:

[0057] - Clue confidence modeling: Title clue o t Precision control factor λ t =0.7*0.85 / (1+0.1)≈0.54, edit distance d(o t q t ) = 2, after normalization d = 0.2, therefore P(o t |g t ,Q,r)=exp(-0.54*0.2) / Z≈0.89 / Z (Z is the normalization constant, and the final value after calculation is 0.82).

[0058] - Single clue relevance probability: Title clue matching degree K(q) t o t )=exp(-γ*0.2 0.6 (Given γ = 0.8, we calculate K ≈ 0.85), therefore P t (rel|q t o t = 0.85 * 0.82 ≈ 0.697; Date clue exact match, P d ≈0.98; exact match for location clues, P e ≈0.98; Visual cues match the image query intent, P v ≈0.75.

[0059] - Multi-clue fusion: Assuming the current weight w t =0.25, w d =0.3, w e =0.3, w v =0.15, global bias term b=0.1, calculated as ∑(w i *logit(P i))≈0.25*logit(0.697)+0.3*logit(0.98)+0.3*logit(0.98)+0.15*logit(0.75)≈2.03, and finally P(rel|s,q)=σ(2.03+0.1)≈0.87.

[0060] 4. Based on the fusion model calculation, P(rel|s,q) = 0.87, ranking first. The system displays the initial selection results to the user. After the user clicks, the original image is displayed, and the relevant areas are highlighted.

[0061] 5. The user clicks and confirms the result is valid. The system records y=1 and updates the weights according to step S204: learning rate η=0.01, w t ←0.25-0.01*(0.87-1)*logit(0.697)≈0.25+0.01*0.13*0.83≈0.251, which increases the title weight. t and location weight w e (The same calculation and update are performed.)

[0062] Figure 3 The user interaction flow process shown in the example fully embodies the core design concept of "tolerance of noise, probabilistic reference, guided discovery, and human judgment".

[0063] (V) Beneficial Effects

[0064] Compared to existing technologies, the beneficial effects of this invention stem primarily from its unique design philosophy: addressing the fundamental dilemma of "irreparable OCR noise" in the digitization of old newspaper documents, it abandons the futile effort of pursuing "correct text" display and instead pursues "effective" discovery. The system is responsible for extracting probabilistic clues from the noise, while the user reads the original image for the final judgment. The beneficial effects are reflected in the following aspects:

[0065] 1. Tolerance to noise: In the data preprocessing stage, 70% of the correct OCR text is modeled. However, this system does not pursue the absolute accuracy of all OCR text. Instead, it uses the probability fusion model described in claim 7 to treat text errors as a measurable noise and incorporates its uncertainty into the correlation calculation. This eliminates the need for manual indexing and modification, saving a lot of manpower and resources in data processing.

[0066] 2. Probabilistic Reference: During the results display phase, the system avoids directly displaying low-quality OCR text that may be full of garbled characters. Instead, it presents users with clear metadata and quantified relevance probabilities (image quality and "hit rate"). This probability value is the non-deterministic "reference" provided by the system.

[0067] 3. Guided Discovery: In the initial selection page stage, metadata records, image quality, and approximation probability are displayed. After the user clicks on the result of interest, the original, unaltered newspaper image is shown to them, thus compensating for OCR errors through human-computer interaction.

[0068] 4. Human Judgment: Users exercise the final "judgment right" by directly reading the original documents, determining whether the result is truly relevant. The user's judgment behavior (such as clicking, dwelling, marking) then flows back to the system through the feedback subsystem, forming an optimized closed loop of "human-machine collaboration".

[0069] 5. Multimodal fusion: For the first time in old newspaper document retrieval, text, visual and structural cues are unified under a probabilistic framework, realizing the natural fusion of "searching for images by text" and "searching for text by images".

[0070] 6. Deployment-friendly: All model parameters can be updated online without reconstructing existing indexes. Combined with containerized deployment and horizontal scaling solutions, this greatly reduces system maintenance costs and ensures high availability. Attached Figure Description

[0071] Figure 1 System overall architecture diagram

[0072] Figure 2 Workflow diagram of the probabilistic fusion sorting engine

[0073] Figure 3 User search human-computer interface

[0074] Figure 4 System data interaction interface relationship diagram

[0075] Figure 5 System deployment architecture diagram.

Claims

1. A multimodal intelligent discovery system for low-quality OCR historical documents, characterized in that, It includes a data preprocessing subsystem, an index storage subsystem, a query processing and sorting subsystem, a human-computer collaborative feedback subsystem, and a deployment and maintenance module. The data preprocessing subsystem is used to perform quality assessment, layout analysis, and multimodal cue extraction on scanned document images. The index storage subsystem is used to build and store multi-clue inverted indexes. The query processing and sorting subsystem is used to parse user queries and calculate the relevance between documents and queries based on a probability fusion model. The human-machine collaborative feedback subsystem is used to collect user feedback and dynamically adjust the weights of the ranking model; the deployment and operation module adopts Docker containerization deployment and Kubernetes service orchestration to ensure system scalability and availability.

2. The system according to claim 1, characterized in that, The data preprocessing subsystem includes: an image quality assessment module, which uses a CNN model (such as ResNet) for end-to-end quality scoring pre-training and outputs the comprehensive quality score Q and grade label of the scanned image; a layout analysis module, which uses a YOLOv8, Faster R-CNN or DETR deep learning model (fine-tuned on an old newspaper dataset) to identify article areas, title areas, image areas and advertising areas in the document; and a multimodal cue extraction module, which is used to extract textual cues (title, date, named entities), visual cues (2048-dimensional deep feature vectors extracted from image areas using ResNet or CLIP-ViT), and structural cues (page layout and area coordinates).

3. The system according to claim 1, characterized in that, The multi-threaded inverted index constructed by the index storage subsystem stores the content of the clue, the coordinates of the relevant document and region, and the initial confidence score of the clue. It adopts a hybrid architecture of Elasticsearch and Milvus, with "document fragments" as the basic index unit, and includes index fields such as doc_id, text_title, and visual_feature.

4. The system according to claim 1, characterized in that, The query processing and ranking subsystem includes a multimodal query understanding module and a probabilistic fusion ranking engine. The multimodal query understanding module is used to parse the natural language or image input by the user into a structured query clue vector. The probabilistic fusion ranking engine is used to calculate the relevance probability between document fragments and queries based on a probabilistic fusion model of multimodal clues, ensuring that 95% of query requests are returned within 500ms and supporting an index throughput of thousands of pages of documents per hour.

5. The system according to claim 1, characterized in that, The feedback signals collected by the human-machine collaborative feedback subsystem include at least one of user click behavior, dwell time, and confirmation or rejection flags. Data is transmitted through the POST / api / vl / feedback interface. The cue weight matrix in the probabilistic fusion ranking engine is dynamically updated using the stochastic gradient descent method, and a learning rate decay and weight change monitoring mechanism is set up.

6. A multimodal cue probability fusion ranking method for the system of claim 1, characterized in that, Includes the following steps: Step S201: Extract cue observations o from the document region r with quality level Q. i Confidence modeling is performed, and its conditional probability model is P(o i |g i ,Q,r)=exp(-λ i (Q, r)*d(o i g i )) / Z, where d(·,·) is the metric function, λ i Z is the precision control factor, and Z is the normalization constant. Step S202: Calculate the single-clue relevance probability based on the inexact matching kernel function K(q) i o i ) and the confidence level of the clues P(o i |g i The product of Q and r); Step S203: Based on the weighted Logit model, fuse the relevance probabilities of multiple clues to obtain the final relevance probability; Step S204: Dynamically update lead weights based on user feedback through an online learning mechanism.

7. The method according to claim 6, characterized in that, The precision control factor λ i The calculation formula is λ i (Q, r)=Q*BaselineAccuracy(r) / (1+HistoricalErrorRate(o i The inexact matching kernel function is defined as K(q) i o i )=exp(-γ*d(q i ,o i ) p ), where p is a smoothing factor with a value range of [0.5, 0.7].

8. The method according to claim 6, characterized in that, The multi-clue probability fusion model is P(rel|q,o)=σ(∑(w i *logit(P i (rel|q i o i )))+b), where σ(·) is the Sigmoid function, logit(x)=log(x / (1-x)), w i Let w be the dynamic weight of clue i, and b be the global bias term; in the online weight learning mechanism, the weight update formula is w. i ←w i -η*(Py)*logit(P i ), where η is the learning rate and y is the actual label provided by the user.

9. The method according to claim 6, characterized in that, Visual cues are extracted using ResNet or CLIP-ViT models to obtain 2048-dimensional deep feature vectors. Textual cues are extracted using OCR tools. Dates are extracted using regular expressions or NER models. Named entities are identified using NER models finely tuned on historical text.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 6 to 9.

Images