Approximate repetition detection method and system fusing local retrieval and multi-dimensional decision

Abstract

The invention discloses an approximate repetition detection method and system fusing local retrieval and multi-dimensional decision. The method comprises the steps that a to-be-detected document is preprocessed, and text content and service meta-information are extracted; utilizing MinHash to generate a compact signature; quickly recalling a candidate set from a massive historical library through an LSH index; adopting a paragraph-level weighted Jaccard algorithm to accurately calculate the structural similarity between the candidate document and the document to be detected; when the similarity exceeds a threshold value, intelligent decision making is carried out according to a preset priority chain considering multi-dimensional business rules such as release mechanism levels and time, and finally reserved documents are determined; and recording the structured audit log in the whole process for tracing. The method realizes approximate repetition detection with high recall, high precision, low time consumption and interpretable and auditing decision process, and is especially suitable for massive de-duplication scenes of strong normative texts such as government affair official documents, news announcements and the like.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, specifically to a data cleaning and document deduplication method, and more particularly to an approximate duplicate detection method and system that integrates local retrieval and multidimensional decision-making. Background Technology

[0002] With the rapid development of e-government, media aggregation, and web crawling technologies, massive amounts of electronic documents (such as government documents, policy announcements, news releases, and research reports) are being aggregated onto various information platforms from multiple channels and sources. During the dissemination, reprinting, and editing processes, these documents often generate a large number of semantically identical or highly similar duplicates or near-duplicates due to format conversion, paragraph reorganization, minor editing, or differences in metadata. This data redundancy not only wastes storage and computing resources but also seriously affects data quality, retrieval accuracy, and the effectiveness of subsequent analysis and decision-making. Existing document deduplication technologies mainly include: Precise hash matching: such as calculating the MD5 or SHA-1 message digest of the full document. This method can only identify completely identical text; it fails to recognize any additions, deletions, space adjustments, or format changes, and cannot identify near-duplicates, resulting in a high false negative rate. Rule-based matching based on key fields: for example, comparing unique fields such as "document number" or "document ID". This method highly relies on the completeness and standardization of fields. In actual multi-source data collection, missing fields and inconsistent formats are extremely common, leading to poor robustness and severe false negatives. Semantic vector-based deep learning methods utilize models such as BERT and Word2Vec to convert documents into high-dimensional semantic vectors, determining duplication by calculating vector similarity. While this method effectively captures semantic similarity, its computational cost for model inference is high, making it difficult to handle large-scale, high-concurrency online processing demands. Furthermore, its decision-making process lacks interpretability, failing to meet the auditing requirements of stringent compliance scenarios. Although publicly available literature includes solutions using MinHash for text deduplication, these primarily focus on general webpage deduplication and have not been optimized for business texts with fixed structures and rich metadata, such as official documents. They also lack the ability to construct decision chains and audit log systems that integrate business rules.

[0003] Therefore, existing technologies struggle to achieve a balance across the five dimensions of high recall, high precision, low computational cost, strong alignment with business logic, and complete auditability. There is an urgent need for an innovative technical solution that can efficiently and accurately identify nearly identical content in massive amounts of documents, make reasonable decisions based on clear and interpretable business rules, and ensure that the entire processing is traceable and auditable. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned defects of the prior art and provide an approximate duplicate detection method and system that integrates local retrieval and multi-dimensional business decision-making.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an approximate duplicate detection method that integrates local retrieval and multi-dimensional business decision-making, comprising the following steps:

[0007] The target document to be tested is preprocessed to extract its standardized text content and business metadata, including the publishing organization level and publication time.

[0008] Based on the standardized text content, a Locality Sensitive Hash Signature (LSH) of the target document is generated as a compact feature representation;

[0009] Using locality-sensitive hash indexing, based on the compact feature representation, a set of candidate approximately duplicate documents can be quickly retrieved from the historical document library;

[0010] For each candidate document in the candidate set of approximately duplicate documents, a structured similarity value is calculated between it and the target document. The structured similarity value is calculated based on the segmentation and matching of paragraphs in the two documents and the assignment of weights to the matched paragraphs.

[0011] If the structured similarity value exceeds a preset threshold, the candidate document and the target document are determined to be an approximately duplicate candidate pair. Based on the preset multi-dimensional business decision rules and combined with the business metadata, the document in the approximately duplicate candidate pair is retained.

[0012] The audit log records include the candidate set of nearly duplicate documents, the structured similarity value, the decision rules used, and the judgment results.

[0013] As a further improvement to the method of the present invention, the step of generating the local sensitive hash signature of the target document specifically involves: using a minimum hash algorithm to represent the standardized text content as a set of elements, and using a set of hash functions to calculate the minimum hash signature vector of the set.

[0014] As a further improvement to the method of the present invention, the fast retrieval using the local sensitive hash index includes: dividing the minimum hash signature vector into multiple sub-bands, calculating the hash value of each sub-band as a bucket key, querying the index structure that stores the correspondence between the bucket key and the document identifier, obtaining the initial candidate document identifier, and filtering documents with low similarity by comparing signatures to form the candidate set of approximately duplicate documents.

[0015] As a further improvement to the method of the present invention, the calculation of the structured similarity value includes:

[0016] The target document and the candidate document are each divided into multiple paragraphs;

[0017] Perform content-based matching on paragraphs of two documents to identify matching paragraph pairs;

[0018] Different weighting coefficients are assigned to different types of paragraphs, with key paragraphs having a greater weight than ordinary paragraphs;

[0019] Based on the weighted Jaccard similarity algorithm, the similarity between two documents is calculated according to the weight of the matched paragraph pairs as the structured similarity value.

[0020] As a further improvement to the method of the present invention, the key paragraph type includes at least one of the following: title paragraph, policy basis paragraph, and responsible entity paragraph.

[0021] As a further improvement to the method of the present invention, the multidimensional business decision rules are an ordered sequence of rules, including at least:

[0022] First priority rule: Compare the publishing organization level of two documents and retain the document with the higher level;

[0023] Second priority rule: When the publishing organization level is the same, compare the publication time of the two documents and keep the document with the later publication time.

[0024] As a further improvement to the method of the present invention, the business metadata further includes file security level and / or document number, and the multidimensional business decision rule sequence further includes:

[0025] Third priority rule: When the publication time is the same, compare the file security level of the two documents and retain the document with the higher security level;

[0026] Fourth priority rule: When files have the same security level, compare the document numbers of the two documents and decide which document to keep based on the numbering rules.

[0027] As a further improvement to the method of the present invention, the recording of audit logs includes: generating a unique task identifier for each detection task, recording the identifier of the candidate approximately duplicate document set, the structured similarity value between each candidate document and the target document, the business element information value involved in the decision, the identifier of the triggered decision rule, and the identifier of the document that is finally retained in a structured data format, and persistently storing the structured data in the audit database.

[0028] Secondly, the present invention provides a near-duplicate document detection system that integrates local retrieval and multi-dimensional business decision-making, comprising:

[0029] The preprocessing module is used to preprocess the target document to be detected, extracting its standardized text content and business metadata.

[0030] The feature generation module is used to generate a compact feature representation of the target document based on the standardized text content;

[0031] The candidate retrieval module is used to retrieve a set of candidate approximately duplicate documents from the historical document library based on the compact feature representation using a locality-sensitive hash index.

[0032] The similarity calculation module is used to calculate the structured similarity value between each candidate document in the candidate approximately duplicated document set and the target document;

[0033] The decision module is used to determine the retention of near-duplicate candidate pairs according to preset multi-dimensional business decision rules when the structured similarity value exceeds a preset threshold.

[0034] The logging module is used to record audit logs.

[0035] As a further improvement to the system of the present invention, it also includes a configuration management module for dynamically configuring at least one of the following parameters: the number of hash functions used by the feature generation module, the local sensitive hash index parameter used by the candidate retrieval module, the paragraph weight used by the similarity calculation module, and the business decision rule sequence loaded by the decision module.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] The combination of high recall and high efficiency is achieved through the first-stage rapid filtering of MinHash+LSH, which can efficiently recall potentially duplicate documents from massive historical databases, maintain millisecond-level response even with large-scale data, and ensure a high recall rate.

[0038] More accurate and robust similarity measurement: The paragraph-level weighted Jaccard similarity algorithm overcomes the shortcomings of traditional MinHash, which ignores text structure. By identifying key paragraphs and assigning them high weights, the similarity calculation better reflects the core semantic overlap of the document and has better robustness to paragraph reordering, addition or deletion of non-critical content, and formatting noise.

[0039] The decision-making process is intelligent and interpretable: a multi-dimensional decision rule chain based on business metadata is introduced, upgrading the simple "similarity judgment" to "intelligent decision-making based on similarity and business rules". The system can automatically make choices that conform to business common sense and norms (such as retaining high-level, new date, and high-confidentiality documents), and the decision-making logic is clear and explicit.

[0040] The entire process is auditable and traceable: Through detailed and structured audit logs, the entire processing chain is made transparent. Any deduplication result can be traced back through the logs to reproduce the candidate set, similarity calculation details, and decision-making basis at that time, which greatly meets the stringent requirements of government affairs, finance, and other fields for data operation compliance, security, and auditability.

[0041] The system is highly scalable and easily integrated: it adopts a modular and loosely coupled design. MinHash signatures, LSH indexes, decision rules, weight parameters, etc., can all be dynamically adjusted through the configuration center, making it easy to customize and optimize for different business scenarios (such as government, media, and enterprises). The system can be easily integrated with existing middleware such as Elasticsearch, relational databases, and message queues, demonstrating good engineering feasibility and scalability. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0043] Figure 1 This is an overall flowchart of the near-duplicate document detection method that integrates local retrieval and multi-dimensional business decision-making provided in the embodiments of the present invention.

[0044] Figure 2 This is a schematic diagram of the fast recall of candidate sets (step S3) and the LSH index principle in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of paragraph weight allocation and calculation in the structured similarity calculation (step S4) of this embodiment of the invention.

[0046] Figure 4 This is a schematic diagram of the rule chain execution logic for multi-dimensional business rule decision (step S5) in an embodiment of the present invention.

[0047] Figure 5 This is an architectural block diagram of the near-duplicate document detection system that integrates local retrieval and multi-dimensional business decision-making, provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The method steps described in the following embodiments are only for explaining the invention and are not intended to limit the invention.

[0049] Example 1:

[0050] This embodiment describes a method for detecting approximately duplicate documents that integrates local retrieval and multi-dimensional business decision-making. It is applied to a provincial government data sharing platform to intelligently deduplicate policy documents, announcements, and notices automatically crawled daily from provincial, municipal, and county-level government websites and mainstream news media before they are stored in the database. Figure 1 As shown, the specific steps of this method are as follows:

[0051] Step S1: Document preprocessing and feature extraction.

[0052] A newly captured official document, titled "Implementation Plan of XX City on Implementing the 'Regulations on Promoting the Digital Economy of XX Province'", is in HTML format.

[0053] 1. Formatting Parsing: Uses a document parser (such as the BeautifulSoup library or the Apache Tika tool) to parse HTML documents, automatically identifying and stripping all formatting errors. <script>、<style>、<header>、<footer>标签以及导航栏、广告、页眉页脚、水印、注释及附件引用等非正文干扰信息，精准提取出文档的标题和核心正文文本流。

[0054] 2.文本清洗与标准化：

[0055] 对提取出的正文进行基于自然语言处理的段落边界检测，按空行、缩进变化或语义连贯性切分为独立的自然段落。

[0056] 对每个段落执行深度清洗操作：

[0057] 字符标准化：统一全角与半角字符，转换中文繁简体为简体。

[0058] 格式归一化：将各类日期表达（如"二零二四年八月十五日”、"2024-08-15”、"2024年8月15日”）统一为"YYYY-MM-DD”格式（例如"2024-08-15”）；将数字、金额等格式进行标准化。

[0059] 停用词移除：加载通用及领域停用词表，移除"的”、"了”、"和”、"在”、"根据”等无实际比对意义的词汇。

[0060] 大小写归一化：将英文字符统一转换为小写。

[0061] 3.业务元信息抽取：所述业务元信息至少包括发布机构级别和发布时间，还可包括文件密级、文档编号等。同时，从文档的元数据属性或通过规则从正文的特定位置（如文头、文尾）提取关键的业务属性字段：

[0062] level_num（发布机构级别）：根据发文单位名称（如"XX市人民政府”），查询预设的行政层级编码映射表，将其转换为整数值。例如，可配置为：国务院=1，省级政府=2，市级政府=3，依此类推。

[0063] publish_date（发布日期）：从正文末尾或特定字段提取，并标准化为"YYYY-MM-DD”格式。

[0064] security_level（文件密级）：识别"公开”、"内部”、"秘密”、"机密”、"绝密”等密级标识，并映射为数值（如0，1，2，3，4）。

[0065] doc_number（发文字号 / 文号）：利用正则表达式从标题或正文开头提取唯一的发文字号，如"X政发〔2024〕12号”。若缺失，则记录为特定标识（如"UNKNOWN”）。

[0066] 输出：经过上述处理后，得到两个核心输出：标准化段落列表 clean_paragraphs（用于后续内容比对）和业务元信息字典 meta_info（用于后续业务决策）。

[0067] 步骤S2：紧凑特征表示生成。

[0068] 本步骤旨在将文档内容转换为一个固定长度、对局部修改鲁棒的数值签名。

[0069] 1.文档表示建模：将步骤S1得到的标准化段落列表 clean_paragraphs = [p1,p2, …, pn] 中的每个段落 pi 视为一个独立的集合元素。整个文档被建模为一个无序的段落集合 D。

[0070] 2.最小哈希（MinHash）签名计算：

[0071] 初始化一个包含 k 个独立哈希函数的哈希族 H = {h1, h2, …, hk}。在本实施例中，设置 k = 128（即num_perm=128）。每个哈希函数 hi 能将任意字符串映射为一个固定大小的整型哈希值。

[0072] 对于文档集合 D 中的每一个段落 p，计算其在所有 k 个哈希函数下的哈希值。

[0073] 对于每一个哈希函数 hi，遍历文档所有段落的哈希值，选取其中的最小值 mi，即：

[0074]

[0075] 将 k 个最小值按序排列，构成该文档的 MinHash签名向量。此向量即为文档的紧凑特征表示，具有以下关键性质：两个文档签名向量对应位置相等的概率，等于它们原始段落集合的Jaccard相似度。

[0076] 步骤S3：候选近似文档快速召回。

[0077] 本步骤利用局部敏感哈希（LSH）技术，在海量历史文档库中快速定位可能与新文档重复的少量候选。

[0078] 所述局部敏感哈希索引采用分桶机制，将签名向量划分为若干子带，对每个子带计算哈希值作为桶键（Bucket Key），用于快速定位候选文档。

[0079] 1.LSH索引构建与查询：

[0080] 索引结构：系统为历史文档库维护一个LSH索引。该索引将每个历史文档的MinHash签名向量 M 划分为 b 个（band）子带，每个子带包含 r 个连续的签名分量（满足k = b * r）。本实施例设 b=16, r=8。

[0081] 桶键计算：对每个子带的 r 个签名值组合计算一个哈希值（如MD5后取模），作为该子带的"桶键”（bucket key）。一个文档的所有 b 个桶键构成了其在LSH索引中的地址。

[0082] 快速检索：对于新文档，以相同方式计算其 b 个桶键。然后并行查询LSH索引中这b 个桶键对应的所有文档ID列表。将这些列表取并集，即得到初始候选文档ID集合。LSH的原理保证了高相似度的文档有极大概率在至少一个子带上拥有相同的桶键，从而被快速召回。

[0083] 2.候选集精炼：初始候选集可能包含一些相似度较低的"噪声”文档。为进一步提升精度：

[0084] 从主存储中批量加载初始候选集中文档的MinHash签名。

[0085] 计算每个候选文档签名与新文档签名的估计Jaccard相似度：

[0086]

[0087] 此计算仅涉及整数比较，速度极快。

[0088] 设定一个宽松的估计阈值（例如=0.5），过滤掉<的文档，形成最终的、高质量的候选近似重复文档集合 C。集合 C 的规模通常可被压缩至几十甚至几条，极大降低了后续精细比对的计算开销。

[0089] 步骤S4：结构化相似度精算。

[0090] 对于候选集 C 中的每一个文档，本步骤进行精细化的、考虑文档结构和语义权重的相似度计算。

[0091] 1.段落对齐：将新文档 Doc_A 与候选文档 Doc_B 的标准化段落列表进行基于内容的模糊匹配。采用TF-IDF算法将每个段落表示为向量，并计算所有段落对之间的余弦相似度。将相似度超过预设阈值（如0.75）的段落对标记为"匹配段落对”，认为它们内容相似或相同。

[0092] 关键段落识别与权重分配（如图3所示）：

[0093] 关键段落至少包括标题段落、政策依据段落和责任主体段落，还可根据业务需求扩展其他段落类型。

[0094] 系统内置一个可配置的关键段落关键词库，用于识别具有特殊重要性的段落类型，例如：

[0095] 标题段落：包含"通知”、"决定”、"公告”、"方案”等词的段落。

[0096] 政策依据段落：包含"根据”、"依据”、"按照”、"参照”等词的段落。

[0097] 责任主体段落：包含"负责”、"牵头”、"组织实施”、"落实单位”等词的段落。

[0098] 根据关键词匹配结果，为每个段落分配一个权重系数 w。例如：标题段落 w=1.5，政策依据段落 w=1.2，责任主体段落 w=1.1，普通段落 w=1.0。

[0099] 3.计算加权Jaccard相似度：

[0100] 基于加权Jaccard相似度算法，计算两个文档的加权交集与加权并集之比，其中加权交集为所有匹配段落对中较小权重之和，加权并集为两文档所有段落按其权重最大值之和，以此作为所述结构化相似度值。

[0101] 设和分别为Doc_A 和 Doc_B 的加权段落集合（每个元素为段落内容及其权重w）， 表示段落 p 的权重。

[0102] 计算加权交集，即对所有匹配的段落对，取其权重较小值之和：

[0103]

[0104] 计算加权并集，即考虑两个文档所有不重复的段落，每个段落取其可能的最大权重（对于共有段落取两者权重的最大值，对于独有段落取其自身权重）之和：

[0105]

[0106] 计算结构化相似度值 Sim：

[0107]

[0108] 该值介于0到1之间，不仅反映了内容的重合度，还通过权重突出了关键内容重合的重要性，对非关键内容的增减和段落重排具有更好的鲁棒性。

[0109] 4.判定近似重复：设定一个业务相关的相似度阈值（例如0.8）。若，则判定 Doc_A 与 Doc_B 构成一对近似重复候选对，进入最终决策环节。

[0110] 步骤S5：多维业务规则决策。

[0111] 当新文档与多个历史文档均构成近似重复候选对时，本步骤依据预设的业务规则链，智能决定最终保留哪一份文档。

[0112] 1.规则链定义：系统预置一个有序的、可配置的多维业务规则决策序列。一个典型的规则链优先级从高到低依次为（如图4所示）：

[0113] R1: 级别优先：比较两文档的 level_num。数值越小代表级别越高。保留级别更高的文档。若级别相同，则进入下一条规则。

[0114] R2: 日期优先：比较两文档的 publish_date。保留发布日期更晚（即更新）的文档。若日期相同，则进入下一条规则。

[0115] R3: 密级优先：比较两文档的 security_level（数值越大密级越高）。保留密级更高的文档。若密级相同，则进入下一条规则。

[0116] R4: 文号优先：比较两文档的 doc_number，可按字典序、录入时间或特定业务逻辑决定保留优先级。

[0117] 2.规则引擎执行：对于每一对近似重复候选对 (Doc_A, Doc_B)，规则引擎依次执行规则R1至R4。一旦某条规则能够明确区分出两者的优先级（即对应的元信息字段值不同），则立即终止后续规则，并输出保留文档和舍弃文档的判定结果。

[0118] 3.冲突裁决与最终状态：若新文档与多个历史文档比较后，自身在某些比较中被保留，在另一些比较中被舍弃，系统默认的冲突裁决策略是：保留所有比较中优先级最高的那份文档。例如，新文档可能因日期新而胜过一个旧文档，但又因级别低而输给另一个高级别文档，则最终保留那个高级别文档。新文档及其他所有被判定为重复的文档，其状态将被更新。

[0119] 步骤S6：审计日志记录。

[0120] 为实现全过程可追溯、可审计，本步骤详细记录关键操作信息。

[0121] 生成任务标识：为每一次去重处理任务生成全局唯一的任务流水号 task_id。

[0122] 结构化日志记录：针对本次任务，生成结构化的审计日志记录。日志至少包含以下字段：

[0123] task_id, timestamp（处理时间戳）

[0124] new_doc_id（新文档标识）

[0125] candidate_set_size（候选集大小）

[0126] duplicate_pairs（一个数组，记录每一对判定为近似重复的文档对详情，包括双方ID、结构化相似度 Sim、双方的业务元信息、触发的规则 triggered_rule、决策结果decision）

[0127] final_action（最终汇总操作，如保留的文档ID、标记为重复的文档ID列表）

[0128] 日志持久化与索引：将上述结构化日志以JSON Lines格式追加写入到文件系统进行长期归档。同时，将日志实时索引到专用的审计数据库（如Elasticsearch）中，并为task_id, doc_id, triggered_rule 等关键字段建立索引，以支持高效的跨维度查询、统计分析和事后追溯。

[0129] 步骤S7：结果存储与系统状态更新。

[0130] 根据最终决策结果，更新系统状态。

[0131] 1.更新主文档库：将被判定为保留的唯一文档，其记录中的状态字段 status 设置为 1（有效）。将被判定为重复的所有文档，其 status 设置为 0（重复 / 无效），并在dup_source_id字段中填入其所指向的保留文档ID。

[0132] 2.更新索引：

[0133] 若新文档被保留，则将其MinHash签名插入LSH索引，以备后续文档查询。

[0134] （可选）同步更新面向业务应用的全文检索索引（如Elasticsearch），将状态为 0的文档过滤或标记为隐藏，确保最终用户检索结果无冗余。

[0135] 至此，一份新文档的智能化去重流程完成。该方法通过"高效召回→精细比对→规则决策→审计追溯”的闭环，实现了高精度、高效率、高可解释性的近似重复文档检测。

[0136] 实施例二：

[0137] 本实施例描述了一种实现实施例一所述方法的系统。如图5所示，该系统采用模块化、松耦合的微服务架构设计，包含以下核心组件：

[0138] 1.文档预处理模块：作为数据入口，负责接收并解析各种格式的原始文档。内部集成 格式解析单元（支持HTML / PDF / DOCX等）、文本清洗流水线 和 元信息抽取单元，输出标准化的 (doc_id, text_content, meta_info) 三元组。

[0139] 2.紧凑特征生成模块：接收预处理后的文本内容，由 MinHash签名计算器 依据配置的参数（如 num_perm）实时生成文档的紧凑签名向量。

[0140] 3.候选集检索模块：该模块是高效召回的核心，包含：

[0141] LSH索引管理器：维护一个高性能的、支持分布式部署的LSH索引。负责签名向量的插入（索引构建）和基于桶键的快速查询。

[0142] 候选集过滤器：对LSH查询返回的初始结果进行基于签名相似度的快速过滤，输出精炼后的小规模候选集。

[0143] 4.结构化相似度计算模块：该模块实现精细化的内容比对，包括：

[0144] 段落对齐器：基于TF-IDF等算法进行段落匹配。

[0145] 权重分配器：根据关键词库识别并分配段落权重。

[0146] 加权Jaccard计算器：依据公式计算结构化相似度。5.多维决策判定模块：内嵌一个可插拔的 规则引擎（如Drools）。该引擎加载由配置中心下发的业务规则链，接收文档对及其元信息，执行规则序列并输出具有明确业务解释的保留决策 (decision,rule_trace)。

[0147] 6.审计日志记录模块：包含 日志结构化封装器 和 持久化接口，负责将处理全链路的上下文信息封装为结构化日志，并写入文件系统及审计数据库，确保操作可追溯。

[0148] 7.配置管理模块：作为一个独立的中心化服务，通过Web界面或API暴露所有关键运行参数，包括但不限于：MinHash参数 (num_perm)、LSH参数 (bands, rows)、相似度阈值(,)、段落权重定义、决策规则链内容等。支持动态更新，实现系统行为的灵活调优。

[0149] 8.存储模块：

[0150] 主文档存储：采用关系型数据库（如MySQL）或文档数据库，持久化存储文档全文、元信息、MinHash签名及状态 (status, dup_source_id)。

[0151] 审计日志存储：采用Elasticsearch作为主存储，提供强大的实时检索与分析能力；同时使用对象存储（如S3）进行日志文件的长期冷备份。

[0152] 该系统各模块间通过轻量级的RESTful API或高性能RPC（如gRPC）进行同步调用，并通过消息队列（如Apache Kafka）进行异步通信（如日志上报、任务触发），从而实现解耦和高可用性。整体架构支持水平扩展，能够平稳应对日均数千万级文档的处理压力。

[0153] 本发明的保护范围不限于上述实施例。 任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，根据本发明的技术方案及其发明构思加以等同替换或改变，都应涵盖在本发明的保护范围之内。例如，MinHash算法可由SimHash替代；LSH索引可替换为其他近似最近邻搜索库；段落权重分配可引入机器学习模型进行预测；决策规则链可根据媒体、企业等不同行业特性完全重构。改动和修改，这些等价形式同样落于本发明所限定的保护范围之内。< / script>

Claims

1. An approximate duplicate detection method integrating local retrieval and multi-dimensional business decision-making, characterized in that, include: Preprocess the target document to be detected, extracting its standardized text content and business metadata, including at least the publishing organization level and publication time. Based on the standardized text content, a Locality Sensitive Hash Signature (LSH) of the target document is generated as a compact feature representation; Using locality-sensitive hash indexing, based on the compact feature representation, a set of candidate approximately duplicate documents can be quickly retrieved from the historical document library; For each candidate document in the candidate set of approximately duplicate documents, a structured similarity value is calculated between it and the target document. The structured similarity value is calculated based on the segmentation and matching of paragraphs in the two documents and the assignment of weights to the matched paragraphs. If the structured similarity value exceeds a preset threshold, the candidate document and the target document are determined to be an approximately duplicate candidate pair. Based on the preset multi-dimensional business decision rules and combined with the business metadata, the document in the approximately duplicate candidate pair is retained. The audit log records include the candidate set of nearly duplicate documents, the structured similarity value, the decision rules used, and the judgment results.

2. The method according to claim 1, characterized in that, The specific steps for generating the local sensitive hash signature of the target document are as follows: using a minimum hash algorithm, the standardized text content is represented as a set of elements, and a set of hash functions is used to calculate the minimum hash signature vector of the set.

3. The method according to claim 2, characterized in that, The method of using locality-sensitive hash index for fast retrieval includes: dividing the minimum hash signature vector into multiple sub-bands, calculating the hash value of each sub-band as a bucket key, querying the index structure that stores the correspondence between the bucket key and the document identifier, obtaining the initial candidate document identifier, and filtering documents with low similarity by comparing signatures to form the candidate set of approximately duplicate documents.

4. The method according to claim 1, characterized in that, The calculation of the structured similarity value includes: The target document and the candidate document are each divided into multiple paragraphs; Perform content-based matching on paragraphs of two documents to identify matching paragraph pairs; Different weighting coefficients are assigned to different types of paragraphs, with key paragraphs having a greater weight than ordinary paragraphs; Based on the weighted Jaccard similarity algorithm, the similarity between two documents is calculated according to the weight of the matched paragraph pairs as the structured similarity value.

5. The method according to claim 4, characterized in that, The key paragraph types include at least one of the following: title paragraph, policy basis paragraph, and responsible entity paragraph.

6. The method according to claim 1, characterized in that, The multidimensional business decision rules are an ordered sequence of rules, including at least: First priority rule: Compare the publishing organization level of two documents and retain the document with the higher level; Second priority rule: When the publishing organization level is the same, compare the publication time of the two documents and keep the document with the later publication time.

7. The method according to claim 6, characterized in that, The business metadata also includes file security level and / or document number, and the multidimensional business decision rule sequence further includes: Third priority rule: When the publication time is the same, compare the file security level of the two documents and retain the document with the higher security level; Fourth priority rule: When files have the same security level, compare the document numbers of the two documents and decide which document to keep based on the numbering rules.

8. The method according to claim 1, characterized in that, The recorded audit log includes: generating a unique task identifier for each detection task, recording the identifier of the candidate near-duplicate document set, the structured similarity value between each candidate document and the target document, the business element information value involved in the decision, the triggered decision rule identifier, and the final retained document identifier in a structured data format, and persistently storing the structured data in the audit database.

9. An approximate duplicate detection system integrating local retrieval and multi-dimensional business decision-making, characterized in that, include: The preprocessing module is used to preprocess the target document to be detected, extracting its standardized text content and business metadata. The feature generation module is used to generate a compact feature representation of the target document based on the standardized text content; The candidate retrieval module is used to retrieve a set of candidate approximately duplicate documents from the historical document library based on the compact feature representation using a locality-sensitive hash index. The similarity calculation module is used to calculate the structured similarity value between each candidate document in the candidate approximately duplicated document set and the target document; The decision module is used to determine the retention of near-duplicate candidate pairs according to preset multi-dimensional business decision rules when the structured similarity value exceeds a preset threshold. The logging module is used to record audit logs.

10. The system according to claim 9, characterized in that, It also includes a configuration management module for dynamically configuring at least one of the following parameters: the number of hash functions used by the feature generation module, the local sensitive hash index parameter used by the candidate retrieval module, the paragraph weight used by the similarity calculation module, and the business decision rule sequence loaded by the decision module.

Images