An electronic system for automated near duplicate detection using locality sensitive hashing (LSH) and corresponding method

Abstract

Proposed is a digital system using Locality Sensitive Hashing (LSH) for near duplicate detection on weighted datasets providing a robust technical solution to the problem by improving (i) efficiency by allowing for fast and accurate identification of near duplicates in large datasets; (ii) accuracy by extending the hashes calculated via 5 the LSH approach with an insurance-specific weight, thus making sure that the concept of similarity between documents is calculated on a technical and industry-relevant basis; (iii) cost reduction by reducing redundant data lowers storage and processing costs; (iv) data integrity by improved accuracy and consistency in records enhance overall data integrity; and (v) regulatory compliance matching by automated data 10 management practices supporting compliance with industry regulations.

Description

[0001] Pl 481 PCOO

[0002] An electronic system for automated near duplicate detection using Locality Sensitive Hashing (LSH) and corresponding method

[0003] Field of the Invention

[0004] The present invention relates to a near duplicate detection system and corresponding automated method for efficient similarity detection in large datasets and / or digital documents. In particular, it relates to an electronic, digital system and a method for automated near duplicate detection using Locality Sensitive Hashing (LSH) structures, and more specifically, to improve data integrity and automated management of large volumes of documents, automatically identifying and eliminating near duplicates. Even more particularly, the present invention also relates to systems utilizing locality-sensitive hashing to normalize textual data of large unstructured data sets which dan technically be used for the automated identification and elimination of near duplicates, which lead typically to redundancy and errors in the technical operation of automated management systems of large volumes of documents.

[0005] Background of the Invention

[0006] Determination of document similarity by computing devices is used to support a variety of technical functionality. In a recommendation scenario, for instance, location of a document by a computing device describing a product or service that has been purchased by a user may be used by the computing device to locate similar documents describing similar products or services. Similarity measures of these documents (e.g., based on matching of product or service descriptions) may then be used as basis to form recommendations by the computing device for the user, such as to recommend one news article based on the content of another news article with which the user has interacted. Similar techniques may be used in marketing scenarios by the computing device to suggest similar advertisements, find related items in a search context, locate similar social network communications, and so forth. Thus, the uses for determinations of document similarity by a computing device may vary as greatly as what is described by the documents.

[0007] Conventional techniques used to determine or measure document similarity are computing resource intensive, which limits availability of these techniques. In one conventional technique, a brute force approach is used by computing devices in which each document is compared to each other document to determine a similarity measure. Therefore, even in instances of one thousand documents, time complexity of such an approach is in the order of a million operations by the computing devices. This may be further complicated by the sparsity of data exhibited by the documents since each document typically includes relatively few of the billions of available words in a human language. Therefore, comparison of each of the available words as part of the billions of available words may consume a significant amount of resources of the computing devices. In instances of one hundred thousand documents, for instance, a forty-node cluster of computing devices may take approximately six hours to compute similarity of the documents to each other. Since the time complexity is quadratic based on the number of documents, this runtime may quickly increase to days for document comparisons of even larger libraries of documents, such as descriptions of games and movies that may exceed millions of documents in typical online scenarios.

[0008] Insurance industry is another example of a field, where systems have to manage vast amounts of data, including policy documents, claims, customer records, and other related information, which may include millions of unstructured data documents and datasets. Duplicates and near-duplicate records can create significant inefficiencies, increase processing times, and lead to inaccuracies and even technical inoperability for automated information extraction and decision-making. It is again to be noted, that traditional systems and methods for duplicate detection are often inadequate for handling large datasets with the required speed and accuracy. Furthermore, they don't consider the insurance specific terminology, providing a "flat" match that does not encompass the different relevancy of the textual tokens in the specific context.

[0009] Another technical field are systems for finding copyright violations and detecting forged images. Near-duplicate document and image detection and subimage retrieval is an important problem with such systems. Also in this respect, the presented system is able to cope with two technical scenarios: finding (potentially modified) copy- righted documents and images and detecting forged documents and images.

[0010] In summary, finding nearest neighbors has become an important operation on databases, with applications to text search, multimedia indexing, and many other technical areas.

[0011] Summary of the Invention

[0012] It is one object of the present invention to provide a system and a method overcoming the deficiencies, as described above. In particular, it is an object of the invention to provide a novel and improved near duplicate detection system for efficient similarity detection in large datasets and / or digital documents, the near duplicate detection system improving data integrity and automated management of large volumes of documents, automatically identifying and eliminating near duplicates.

[0013] According to the present invention, these objects are achieved, particularly, with the features of the independent claims. In addition, further advantageous embodiments can be derived from the dependent claims and the related descriptions.

[0014] According to the present invention, the above-mentioned objects are solved by the inventive system and method for near duplicate detection using Locality Sensitive Hashing (LSH) providing enhanced operational efficiency, minimized use of resources and thus reduced costs, and improved data integrity within large dataset and digital document handling not only in the insurance industry but in the whole digital technology. By providing a novel automated and technically less error prone system, i.e. more precise and reliable system, in the sense of detected false-true and false-false the present digital system, data processing technology, in particular data processing technology in the automated risk mitigation and (re)insurance field with extremely large and heterogenous datasets (>5mn), is enabled in a technical new way to effectively manage extremely large volumes of digital documents and heterogenous datasets, autonomously and reliably identifying and eliminating near duplicates that lead to redundancy and minimizes operational errors.

[0015] According to the present invention, the above-mentioned objects for the electronic near duplicate detection system for efficient similarity detection in large datasets and / or digital document and corresponding digital method are achieved, particularly, in that the near duplicate detection system receives an input specifying a similarity threshold via an interface, the similarity threshold defining a minimum number of a plurality of buckets where two of respective datasets are measured to be similar, if for the two of respective datasets the minimum number for the same buckets is detected to be equal or above the similarity threshold; in that the near duplicate detection system assesses a data storage of the system comprising the large datasets stored by means of data structures of the data storage and, by means of a tokenizer, to cut each dataset in unique overlapping substrings each of which being composed of contiguous subsequences of tokens within the respective dataset; in that a plurality of weight vectors are generated for the plurality of datasets from the unique overlapping substrings by applying hashing to the unique overlapping substrings creating multiple hashes for the unique overlapping substrings, and by weighting the generated hashes by assigning technologic field specific weights to the hashes and providing weighted hashes resulting in said weight vector for each dataset, wherein the plurality of weight vectors result in a reduction of dimensionality of the plurality of datasets; in that the near duplicate detection system generates, by means of a locality-sensitive hashing unit, a plurality of band hash signature values by further reducing the dimension of the weight vectors applying locality-sensitive hashing to the weight vectors and generating reduced weight vectors comprising weighted single hashes to form band hash signature values from a band of weighted vector elements of the weight vectors as aggregated weighted hashes; in that the near duplicate detection system assigns, by means of a locality-sensitive hashing unit, datasets having same or corresponding hash values to the same bucket of datasets; and in that the near duplicate detection system provides similarities by matching the respective unique overlapping substrings and contiguous subsequences of tokens and mark two datasets as measured to be candidate pairs, if for the two of respective datasets the minimum number for the same buckets is detected to be equal or above the similarity threshold. In the state of the art, locality sensitive hashing (LSH) is a technique used in approximate nearest neighbor (ANN) search. Though, developing digital engines for efficient similarity search is a fast-evolving field, there are still many technical drawbacks. There is a large demand on improved for near duplicate detection systems, triggered by big technology players in the digital field like Google, Netflix, Amazon, Spotify, Uber, and more, all relying on efficient similarity searches for many of their core functions. For example, Amazon uses similarity searches to compare customers, finding new product recommendations based on the purchasing history of their highest- similarity customers. Another example is Google, where every time a user uses Google, a similarity search is performed between a user's query / search term — and Google's indexed internet. A further example is Spotify. Spotify only manages to recommend good music, because their similarity search algorithms are successfully used for matching a user to other customers with a similarly good (or not so good) taste in music.

[0016] However, also the big field of advanced digital risk-focused technologies, used in digital risk mitigation technology with typically millions of datasets to be processed is a suitable field for LSH-based data processing technologies since many risk-related data management and data processing problems technically involve similarity search challenges. Examples are: (i) Redundant datasets, in particular in large heterogenous data accumulations: Near-duplicate records increase storage need and costs and complicate data management; (ii) Processing inefficiencies: Handling redundant records increases processing times and energy consumption for large datasets, as e.g. claim datasets or the like; (iii) Error prone: Inconsistent data leads to errors in the data processing, especially in large datasets e.g. given during underwriting, claims processing, and customer service, and (iv) Regulatory compliance: Ensuring data integrity and accuracy is critical for regulatory compliance. Monitoring and efficiently controlling compliance in millions of digital documents and datasets is an extreme technical challenge. The proposed system has inter alia the advantage, that it allows to efficiently apply a novel way of Locality Sensitive Hashing (LSH) for near duplicate detection on weighted datasets providing a robust and technically reliable solution. Important aspects of the advantages are improvement by (i) Efficiency: The applied novel LSH scheme provides fast and accurate identification of near duplicates in large datasets; (ii) Accuracy: The inventive system extends the hashes generated via the novel LSH approach with a filed-specific weight, as e.g. a risk-specific weight, thus making sure that the concept of similarity between the datasets and / or digital documents is generated on an technology-relevant basis; (iii) Storage / energy / cost reduction: Reducing redundant data lowers storage, processing costs, and energy consumption; (iv) Data integrity: The improved accuracy and consistency in digital records enhance the overall data integrity and consistently applies to the whole large dataset; and (v) Regulatory compliance: The improved data management and digital processing facilitates support compliance with industry regulations or technical safety standards.

[0017] In an embodiment variant, the near duplicate detection system further generates a first plurality of weights for a plurality of the unique overlapping substrings by applying at least one locality-sensitive hashing schemes including at least an element of randomness, and generates a second plurality of weights for the plurality of the unique overlapping substrings using at least one different locality-sensitive hashing schemes, and matches the datasets by all pairs of weights at least in one position, to generate candidate pairs, and measures similarity between corresponding parts of the datasets to filter for false positives generated by the automated method for the near duplicate detection system.

[0018] In another embodiment variant, the near duplicate detection system further generates a filtered set of datasets by removing datasets from the plurality of datasets that are not detected to be similar based on the similarity threshold, and generates, by means of a filter unit a plurality of partitions from the filtered set of datasets resulting for each partition in disjoint datasets, each disjoint dataset and partition being assigned to a different originator unit, and assigns the plurality of partitions to generate candidate pairs for determining dataset similarity of respective datasets of the filtered set of datasets to each other, within respective said partitions.

[0019] In even another embodiment variant, the near duplicate detection system controls a determination of document similarity measure. For example, the dimensionality of the documents is reduced through computation of weight data by a data processing unit of the system, e.g., via a hashing technique such as "minhashing" or the like which is also known as minwise independent permutations locality sensitive hashing. From this wight data, locality sensitive hashing structures providing another hashing technique are used to determine similarity of the wight data to each other, such as by hashing values of the weight data into respective buckets, e.g., collections of similar weight data. Identification of disjoint sets of data is then used as a basis to partition the documents for determination of document similarity by respective nodes of a plurality of computing devices. In this way, an amount of data shuffling between the nodes of the computing devices as part of the determination of document similarity may be reduced. In another example, a weighting is applied by the computing device to attributes of documents as part of the determination of document similarity, e.g., to apply different weights to risk actors than directions for documents.

[0020] In an embodiment variant, the near duplicate detection system further processes each dataset of the large datasets by cutting overlapping substrings of a fixed, predefined size, and hashes similar input items into same buckets having a probability within the similarity threshold by conducting min-wise independent permutations locality sensitive hashing. The hashing can e.g. be applied by using a plurality of different hash scheme or the hashing can e.g. be applied by using only a single hash scheme.

[0021] In another embodiment variant, the near duplicate detection system further reduces the dimension of the weight vectors by applying said locality-sensitive hashing creating a single hash form a band of nearby elements in the weight vector for generating the bandhash vector.

[0022] Finally, in a further embodiment variant, the unique overlapping substrings is cut into substrings of fixed size, the minimum size being at least 3 or greater.

[0023] Again, the present invention has, inter alia, the advantage that large amounts of datasets, as in particular digital reinsurance or insurance documents, are typically lengthy, legally intricate, and contain clauses with subtle variations that can significantly impact financial outcomes. For instance, two digital documents can be very different considering the simple word overlap method, but are to be considered as near-duplicate in a reinsurance context, and vice versa. Traditional LSH methods, designed for generic document sets, lack the precision needed to capture these aspects. By incorporating field-specific terminologies, as e.g. risk-specific or (re) insurance-specific terminologies, semantic context, and industry-tailored optimizations, the present inventive system ensures higher accuracy and scalability. The present invention provides (i) domain specific Feature engineering, including (re)insurance terminology and ontologies, and in particular a custom preprocessing step that performs tokenization and stemming considering reinsurance specific "tokens", and structure detection, (ii) a hashing system, using context-aware hashing, as classification of sections by content (such as policy clauses, exclusions) and structure: (tables, notes), where different weights are assigned to hashes according to the section they belong to, and (iii) "semantic" LSH, where weighted hashes enhance the ability to capture the significance of different parts of a document, thereby improving the detection of semantically similar content.

[0024] In summary, there are, inter alia, three important technical advantages of the present inventive system over all available prior art systems: (1 ) Domain specific feature engineering: The inventive system allows to incorporate and to take into account technical filed specific terminology and ontologies, e.g. (re)insurance. The inventive system allows to incorporate and to take into account a custom preprocessing step that performs tokenization and stemming considering field specific "tokens". Further the inventive system allows to incorporate structure detection; (2) Hashing system: The inventive system allows to incorporate and to take into account context-aware hashing. The inventive system allows to incorporate and to take into account classification of sections by content (such as policy clauses, exclusions) and structure: (tables, notes) . The inventive system allows to assign different weights to hashes according to the section they belong to; and (3) "Semantic" LSH: The inventive system allows to incorporate weighted hashes, the weighted hashes enhancing the ability to capture the significance of different parts of a document, thereby improving the detection of semantically similar content.

[0025] The description herein, in particular with regard to the drawings below, is presented for purposes of illustration and understanding, and is not intended to be exhaustive or limited to the invention in the form disclosed. Various modifications and variations will be apparent to those of ordinary skill in the art. The embodiments, in particular with regard to the drawings, are chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as suited to a particular contemplated use. According to on embodiment variant of the present invention, there is provided a computer-readable storage medium comprising computer-executable program instructions stored thereon that when executed by a computer processor perform a method according to one of the aspects of the present invention.

[0026] Brief Description of the Drawings

[0027] The present invention will be explained in more detail below relying on examples and with reference to these drawings in which:

[0028] Figure 1 shows a schematical illustration of an exemplary architecture of the inventive automated near duplicate detection system 1 and corresponding automated method for efficient similarity detection in large datasets and / or digital documents 141 . The reference number shows the similarity threshold 1 1 received via the interface 13 and defining the minimum number 121 of buckets 12. Two of respective datasets 141 are measured to be similar, if for the two of respective datasets 141 the minimum number 121 for the same buckets 12 is detected to be equal or above the similarity threshold 1 1 . Reference number 14 shows a data storage 14 of the system 1 storing the large datasets 141 by means of appropriate data structures. Reference number 15 shows the slicing module 15 cutting 21 each dataset 141 in unique overlapping substrings 151 each of which being composed of contiguous subsequences of tokens 152 within the respective dataset 141. Reference number 16 shows the hashing unit generating a plurality of weight vectors 161 from the plurality of datasets 141 and unique overlapping substrings 151 by applying hashing 22 to the unique overlapping substrings 151 and creating multiple hashes resulting in a weight vector 161 for each dataset 141 . The weight vectors 161 creating a reduction of dimensionality of the plurality of datasets 141 providing similarities by matching the respective unique overlapping substrings 151 and contiguous subsequences of tokens 152. Reference number 17 shows the locality-sensitive hashing unit generating the band hash signature values 171 further reducing the dimension of the weight vectors 161 by applying locality-sensitive hashing to the weight vectors 161 and generate reduced weight vectors 171 comprising single hashes 172 as band hash signature values 171 from a band of vector elements of the weight vectors 161 . the locality-sensitive hashing unit 17 also assigns the datasets 141 with same or similar hash values 172 to the same bucket 12. The reference number 19 shows the threshold trigger unit, wherein detecting the minimum number 121 for the same buckets 12 to be equal or above the similarity threshold 1 1 , these two datasets 141 are marked of flagged as measured to be candidate pairs 1 12, i.e. near duplicate of each other's. Thus, the detection of the minimum number 121 for the same buckets 12 to be equal or above the similarity threshold 1 1 for two selected datasets 141 triggers the marking of the two datasets 141 as measured or detected to be candidate pairs 1 12.

[0029] Figure 2 shows a schematical illustration of an exemplary traditional LSH approach for content-agnostic near-duplicate document clustering. In many industrial field, inter alia to the development under industry 4.0, vast amounts of datasets and digital documents are accumulated. This is specifically true for the risk-transfer and riskmitigation industry. Furthermore, the datasets and digital documents collected are often lengthy, legally intricate, and contain clauses with subtle variations that can significantly impact outcomes. For instance, two documents can be very different considering the simple word overlap method, but are to be considered as nearduplicate in a (re)insurance context, and vice versa. Traditional LSH methods, designed for generic document sets, lack the precision needed to capture these aspects. By incorporating reinsurance-specific terminologies, semantic context, and industry- tailored optimizations, this innovation ensures higher accuracy and scalability.

[0030] Figure 3 shows a schematical illustration of an exemplary system with k =4, m =4, L =6: sharing of the first-level partition among the different hash functions. Hash tables (ui, U2), (ui, U3) and (ui, U4) are first partitioned according to ui. Each partition (shown in different grey-levels) is then partitioned according to hash functions U2, U3 and u4. The corresponding hash values share the results of the first-level partitions among the different hash functions. For instance, consider the hash tables for gi,2= (ui, U2) and gi,3 = (ui, U3). Both these tables are partitioned first using the same hash function ui. Instead of repeating this partitioning work, instead the first level partitioned table among these hashes can be shared. Figure 3 shows an example of this sharing. In order to achieve this, a 2-step process can be used: the data are portioned on the basis of the m hash functions ui...um. This step only involves m partitions. In the second step, each first level partition are taken, say Ui, and partition it on the basis of Ui+i...um. This involves a total of L = m / 2 partitions. This reduces the number of partitioning steps from 2

[0031] * L (first and second level partitions) to L+m, which takes much less time since m ~ O(^L) .

[0032] Figure 4 shows a diagram illustrating an exemplary estimated versus actual runtimes for LSH creation and querying (1000 queries) . It is to be noted, that, as figure 4 illustrates, the LSH creation and querying of the inventive step can be also used as parallel LSH, involving parallel computing nodes. This allows technically to scale the inventive system and optimize the required execution runtime.

[0033] Figure 5 shows a diagram illustrating an exemplary LSH structure according to the invention aiming to place similar values into the same buckets.

[0034] Figure 6 shows a simplified flowchart illustrating an exemplary classification of the hashes 163 and appropriate weight assignment process comprising 5 exemplary steps, the weights being static, and / or learned, and / or rule-based: Step 1 includes start processing: The system starts by processing a hash 163 of the field-specific dataset and digital document 141 . In step 2, the system performs keyword checking: The system checks whether keywords or patterns related to earthquake clauses are present. In step three, the system performs classification: The section can e.g. be either classified as "earthquake" or "non-earthquake" based on the presence of these keywords. In step 3, the system assigns the generated weights. If it's an earthquake-related hash 163, the weight can be assigned dependent on the used approach (static, learned, or rulebased) . If it's a non-earthquake hash, a lower weight can e.g. be assigned. In step 5, output is generated by the system. The system assigns to each hash 163 the weight and can e.g. output the criticality (e.g., critical or non-critical) and the associated weight for each hash 163. In summary, in this embodiment variant, the technical approach of using static weighting assigns fixed numeric weights based on predefined rules, learned weighting applies a machine learning model structure to learn to assign weights dynamically to the hashes 163 based on historical data, and rule-based weighting uses rules based on keywords, clause structures, or other patterns determine the weight. The inventive technical framework allows to process and classify field-specific clauses (in the example above "earthquake deductible clauses"), while adjusting the criticality based on static rules, learned patterns, or specific clauses defined by the user. Figure 7 shows a simplified flowchart illustrating exemplary the four main processing steps 21 24 of the process steps 2 of the near duplicate detection system 1.

[0035] Detailed Description of the Preferred Embodiments

[0036] In the drawings, like reference numerals designate identical or corresponding parts throughout the several views. Further, as used herein, the words "a," "an" and the like generally carry a meaning of "one or more," unless stated otherwise. Furthermore, the terms "approximately," "approximate," "about," and similar terms generally refer to ranges that include the identified value within a margin of 20%, 10%, or preferably 5%, and any values therebetween.

[0037] Figure 1 schematically illustrates a block diagram of a near duplicate detection system 1 for efficient similarity detection in large datasets 141 of documents, the detection system 1 . It is to be noted, that the terms "near duplicate", "semantic similarity", "bucket" may be used more or less interchangeably, in the following. Determination of similarity of documents by a data-processing device (e.g., digital or electronic documents) is a computationally expensive operation, especially when confronted with highly sparse data included in a multitude of documents, in particular unstructured documents and data sources. Such large datasets and digital documents are typical, for example in sensor-data-dense smart technologies or also in riskmitigation and insurance industry. For example in the (re)insurance filed, the technical peculiarities of (re)insurance contracts are particularly in respect to length, hierarchical clause references, high liability if wording differs by a single phrase. In particular reinsurance contracts, especially treaty agreements, are legally dense, hierarchically structured, and extremely sensitive to wording nuances. These characteristics make them technically especially challenging to parse, normalize, and automate in comparison to standard insurance policies or general legal documents. Thus, such fieldspecific technical peculiarities are most relevant when designing automated systems to work with data processing of e.g. reinsurance contracts, since (i) reinsurance contracts are long, often spanning dozens to hundreds of pages, (ii) reinsurance contracts consist of multiple sections: declarations, general conditions, specific clauses, schedules, endorsements, annexes, slip wording, etc., and (iii) some content may be embedded in attachments (e.g., rate-on-line sheets, treaty wordings, appendices) . These characteristics have a big technical impact on how such a system can be realized: (i) Parsing engines must support multi-document and multi-section extraction, (ii) the system requires document segmentation: breaking into meaningful components (e.g., clause headers, subclauses), and (iii) there is a need for OCR, PDF table parsing, and layout-aware models for scanned treaties. Other field-specific characteristics of reinsurance contracts are the (a) clauses reference each other cross-sectionally, often in a deeply nested hierarchy (e.g. "Subject otherwise to clause 4(a) (ii)" - "Notwithstanding anything contained in paragraph B.3.4...") . Some clauses can override or modify others later in the document. This requires a graph-like representation of clauses, not flat text. The system 1 must track: Clause IDs and hierarchy, Override relationships, and Conditional references ("only if Clause 5.2 applies"). Further crossreferencing resolution is required to fully interpret a clause's actual impact; (b) There is a high sensitivity to wording differences. A single word or phrase (e.g., "occurrence" vs. "event", or "and" vs. "or") can materially change liability. For example, "...per event and in the aggregate" vs. "...per occurrence, not in the aggregate can shift millions in exposure. As another example, words like "subject to", "notwithstanding", "unless otherwise stated" are legal modifiers that can flip interpretation. Thus, the system 1 must be able to perform fine-grained semantic analysis, detect subtle modifiers and phrase shifts, and / or apply linguistic pattern recognition to infer legal logic. Further, embedding-based similarity models must be precision-tuned, not just general-purpose. Finally, the system can e.g. require clause fingerprinting and sensitive phrase detection pipelines; (c) reinsurance contracts often use custom and free-form wording. I.e. reinsurance contracts are often bespoke, even within the same LoB. Brokers or cedents may negotiate custom clause wording, even for standard concepts like deductibles or retentions. Unlike primary insurance, there exist no universal templates. This has a large technical impact to the system 1 . The system 1 cannot rely on exact match or even template-based matching, clause normalization can e.g. require flexible NLU models trained on reinsurance-specific text, and a feedback loop can be needed for new clause variants to continuously improve models; (d) reinsurance contracts can comprise multilingual and Jurisdictional variation. International treaties can be in multiple languages or follow jurisdiction-specific standards (e.g., London Market vs. US vs. Asia) . The legal interpretation can depend on governing law clauses. The technical impact to the system 1 is that e.g. NLP must support multilingual clause understanding. The system 1 can e.g. require legal-domain translation models, and jurisdiction must be a feature in the clause logic of the system 1 ; (e) reinsurance contracts often comprise embedded financial logic. Clauses often contain embedded formulas, threshold logic, or tiered limits. E.g., "5% of TIV, subject to a minimum of USD 500,000 and a maximum of USD 5,000,000 per occurrence". Financial expressions may be textual, tabular, or semistructured. Thus the system 1 needs hybrid NLP and rules engine to extract quantitative structures, tables must be parsed as semantic clauses, not just layout elements, and clause financials must map to calculable schema for exposure modeling; (f) reinsurance contracts can comprise multi-peril and multi-trigger clauses. Clauses may cover multiple perils with different terms, e.g. "For earthquake: 5% TIV; for wind: USD 1 M flat deductible". Further, trigger conditions may vary within a clause. The technical impact is that the clause extraction of the system 1 must be able to split multi-peril content into separate normalized units. Teh system 1 must be able to handle nested logic, as e.g. "If EQ, then X; else if Wind, then Y; else Z". In summary, often there are fieldspecific requirements of the large datasets 141 to be processes, resulting in the above example in the technical requirement that the system 1 requires domain-trained model structures, clause knowledge graphs or schemas, context-aware clause disambiguation, and semantic and syntactic parsing at high precision. Other fieldspecific requirements given by the characteristics of the large datasets 141 to be processes can e.g. vary. Thus, the system 1 must be able to cope with these characteristics and their technical impact. Thus, the present system 1 is applicable cross-industry. Unseen the above and following re-insurance examples, it is to be stated expressly that the inventive technique can be applied to any other long-form, clause-structured documents such as construction contracts or pharmaceutical licenses etc.

[0038] In is to be noted, that there are two technical main factors that contribute to the inefficiency in conventional techniques, as prior art system disclosed by shingled-minhash for patent deduplication (IBM) or Doc2Vec + LSH for contract comparison, which are size and dimensionality. The inventive system 1 with its section-specific weights go far beyond such uniform shingles and word-vector approaches. Above, size refers to a number of items that are included in a corpus, i.e., the number of documents. Dimensionality refers a number of different elements included in each of the documents, e.g., known words in a language. Accordingly, comparison of a large number of elements (e.g., words) in a large number of documents may cause consumption of significant amounts of computing resources, e.g., even up to days of processing for large online repositories of songs, application stores, social network posts, and so forth.

[0039] In the inventive system 1 , the near duplicate detection system 1 receives a data input specifying a similarity threshold 1 1 via an interface 13 connected to a data transmission network 3, for example, from the technologic field units 5. The similarity threshold 1 1 can e.g. also be inputted by an operator to or defined in the near duplicate detection system 1 , itself. The units 5 can be processing units specific to any technological field, as e.g. industrial plant specific units 51 , monitoring or sensor system specific units 52, in particular smart technology based systems, and / or risk-mitigation or risk-transfer or risk-monitoring system specific processing units 53, or any other units 5i from an industrial field. The similarity threshold 1 1 defines a minimum number 121 of a plurality of buckets 12. Two of respective datasets 141 are measured to be similar, if for the two of respective datasets 141 the minimum number 121 for the same buckets 12 is detected to be equal or above the similarity threshold 1 1 .

[0040] The near duplicate detection system 1 assesses a data storage 14 of the system 1 comprising the large datasets 141 , e.g. the large datasets (141 / 5ii) of one or a plurality of field-specific units 5 / 5i stored by means of data structures of the data storage 14 and, by means of a tokenizer or slicing unit 15, to cut 21 each dataset 141 in unique overlapping substrings 151 each of which being composed of contiguous subsequences of tokens 152 within the respective dataset 141 .

[0041] The tokenizer 15 conducts therefore pre-processing step, specifically designed to capture unit-specific characteristics. For example, the (re) insurance industry typically uses specialized terminology and language terms that may differ significantly from other industry fields. To effectively capture these nuances, the tokenizer 15 provides custom pre-processing steps. This involves developing a domainspecific of field-specific tokenizer 15 that automatedly recognizes unit-specific, e.g. (re) insurance-specific, "tokens", such as policy clauses, endorsements, and specific legal terms, and processes them accordingly. Additionally, a stemming process tailored to unit-specific of field-specific (e.g. (re)insurance-specific) language ensures that variations of terms (e.g., "insure," "insured," "insurance") are consistently treated, preserving the semantic meaning during analysis. The tokenizer 15 also comprises unit- specific structure detection. For example, digital (re)insurance documents often include complex structures such as tables, notes, and appendices, which may not be directly relevant to the document's semantic content. The structure detection step identifies and classifies these sections, allowing them to be demoted or excluded from the similarity analysis. By focusing on the most semantically significant parts of the document such as the main body of the contract or key clauses, this approach ensures that irrelevant sections do not influence the similarity detection process.

[0042] The near duplicate detection system 1 and / or the tokenizer 15 comprises further a weighting unit 155 generating or capturing technology field specific weights 1551 . Typically, all different technical fields use their specialized terminology. For example, the (re)insurance industry, and the technical means used in these fields, uses specialized terminology and legal language that differ significantly from other industries. To effectively capture these nuances, a custom pre-processing step is implemented. This involves developing the domain-specific tokenizer 15 that recognizes technological field-specific "tokens", such as policy clauses, endorsements, and specific legal terms in the risk mitigation industry and processes them accordingly. Additionally, a stemming process tailored to a specific technological field's language ensures that variations of terms (e.g., in (re) insurance technology: "insure, " "insured," "insurance") are consistently treated, preserving the semantic meaning during analysis. Further, technology-field- specific digital datasets 141 or digital documents (as e.g. (re)insurance datasets) often include complex structures such as tables, notes, and appendices, which may not be directly relevant to the dataset's 141 or digital document's semantic content. The structure detection step of the near duplicate detection system 1 identifies and classifies these sections, allowing them to be demoted or excluded from the similarity detection. The structure detection is focused on the most semantically significant parts of the document such as the main body of the contract or key clauses. This technical approach ensures that irrelevant sections do not influence the similarity detection process.

[0043] In detail, for the tokenization, the tokenizer 15 first tokenizes each dataset and digital document 141 , paying special attention to field-specific terminology (e.g. (re) insurance-specific terminology). This involves splitting the digital document 141 into meaningful tokens, such as words or phrases, considering the domain-specific vocabulary. These terms can be white-listed and previously collected parsing a corpus of editorially selected documents with techniques such as Inverse Document Frequency to add a weight 1551 to the tokens. For the section identification and classification: within each digital document 141 , the tokenizer 15 Identifies and classifies different sections of the digital document 141 , such as contract clauses, exclusions, endorsements, tables, and notes. This step is technically crucial because different sections will be weighted differently in the hashing process 16 / 17. The classification process can e.g. be done using a classification system, that relies on a support vector machine classification structure. In other word, there is a domain-specific pre-processing of each dataset and digital document 141 , where the tokenizer 15 stems tuned, for example, to field specific policy language, detection of LOB (Line Of Business), specific section headings or clauses, etc.) by acting as a field-specific (e.g. (re) insurance-specific) parsing module.

[0044] To provide a field-specific example, in a first step, the tokenizer 15 identifies and normalizes earthquake deductible clauses by conducting first text cleaning by remove irrelevant sections (headers, footers, page numbers, etc.) and correct OCR errors, if applicable. Then, the tokenizer 15 applies a segmentation process by break the document into smaller, manageable chunks, for example by using natural language processing (NLP) techniques to split text based on paragraphs, sections, or sentence boundaries. In a second step, the tokenizer 15 performs a pattern recognition for clause identification. In the example of earthquake deductible clauses, those clauses usually contain specific patterns or keywords that are used by the tokenizer 15 to identify them. For example, some common features may include: (a) Keywords / Terms: Earthquake, deductible, limit, threshold, loss, risk, coverage, etc., (b) Numerical Data: Earthquake deductibles often have monetary values, percentages, or other numerical indicators, and (iii) Clause Structure: Earthquake deductible clauses may follow a typical legal format, often detailing the deductible percentage, coverage limits, and conditions. The parsing module can e.g. use regular expressions (regex) or more advanced NLP techniques like Named Entity Recognition (NER) or rule-based matching to locate clauses related to earthquake deductibles. In a third step, field normalization is performed, i.e. after identifying relevant clauses, the next step is to normalize the extracted data into a structured format. Normalization means, in this context, converting the extracted fields into consistent units, types, and categories that are useful for further analysis. The processing may include: (a) Extracting the key fields: For each identified clause, extract specific fields like: Deductible percentage or value, coverage limit, loss threshold, and / or earthquake-related conditions (e.g., magnitude, area, event type), (b) Normalizing units: Ensuring that numerical values are standardized (e.g., converting currency amounts into a common format, converting percentages to decimal form, etc.), (c) Field Mapping: Map the extracted fields into standardized templates, using labels like earthquake_deductible_percentage, coveragejimit, earthquake_threshold, etc., (d) Validating: Use validation rules to ensure that extracted values are plausible. For example, ensure that a deductible percentage lies within a reasonable range (0-100%) . In a fourth step, the tokenizer 15 applies contextual understanding. Reinsurance documents are legal and technical in nature, meaning they often contain complex, context-dependent language. To ensure the module properly identifies the deductible clauses by: (a) using NLP Models: Leverage pre-trained language models or custom-trained models on legal documents to interpret clauses better, and (b) contextual matching: the model structure is built to recognize relationships between fields (e.g., deductible percentage might depend on a defined threshold, or there may be exceptions for certain types of events) . In a fifth step, the tokenizer 15 handles variability in clause wording: The wording and structure of clauses can vary greatly between documents, so it's important, that the tokenizer 15 is technically enabled to account for this variability, e.g. using: (a) synonyms / variants: Earthquake clauses may use different wording, such as "seismic loss," "quake deductible," or "earthquake-related coverage threshold." The tokenizer 15 must be able to detect and technically handle these variations, and (b) clause positioning: Earthquake deductible clauses may appear in different sections of the document (e.g., "Deductibles," " Exclusions," or "Coverage Terms") . The parser module and / or tokenizer 15, thus, scan through multiple sections by identifying the correct location. In a sixth step, modeling may be performed, using e.g. (a) rule-based systems: A rule-based system using regular expressions and logic can identify specific phrases and extract relevant data points, (b) machine learning: Supervised learning can e.g. be applied by training a model on labeled data (where clauses are annotated with deductible information) . A model structure like a sequence labeling algorithm (e.g., CRF, BiLSTM- CRF) or even a transformer-based model (e.g., BERT or T5) can e.g. be trained to recognize and extract relevant clause fields, and / or (c) document classification: If the tokenizer 15 processes many different types of clauses (e.g., hurricane deductibles, flood deductibles), document classification techniques can help separate and focus on earthquake clauses specifically. In a seventh step, the output is structured and prepared for storage. For example, once normalized, the extracted data can be structured in various formats, by the tokenizer 15, such as: (a) JSON or XML: Structured data that can be transferred and consumed by associated downstream applications, (b) assessing databases: The tokenizer 15 may store the data in a relational or NoSQL database for querying and reporting, and / or (c) generating respective tables / spreadsheets, e.g. if the data is needed in tabular form for reporting or analysis. For example, if there is an input clause, as "The Earthquake deductible shall be 5% of the total loss, subject to a minimum of $100,000 and a maximum of $500,000. This deductible applies in the event of an earthquake with a magnitude of 6.0 or greater.", the tokenizer 15 and / or parsing module can e.g. generate a normalized output, as json { "earthquake_deductible_percentage": 0.05, "coverage_limit_min": 100000, "coverage_limit_max": 500000, "earthquake_threshold_magnitude": 6.0 }

[0045] In a last step, the tokenizer 15 and / or parsing module can e.g. be tested and iterated by (a) performing testing on varied document samples: Reinsurance documents are highly variable, so the tokenizer 15 and / or parsing module should be tested on a large dataset with diverse phrasing and structures, and (b) applying an appropriate error handling: The tokenizer 15 must be to handle ambiguous cases and flag instances where it cannot clearly identify the deductible clause.

[0046] By following such a structured, field-specific approach, as illustrated exemplarily by the above example, the tokenizer 15 and / or parsing module is enabled by technical means to efficiently identify, extract, and normalize earthquake deductible clauses in the field-specific dataset and digital document 141 (here e.g. (re)insurance documents), enabling technically automated analysis, comparison, and compliance checks. The example above is a worked example how "Earthquake deductible clause" is identified and normalized by the system and / or tokenizer 15, respectively.

[0047] The feature extraction is done with the weights 1551 . Id est, the weights 1551 are assigned to sections: Based on the classification, the system 1 assign weights 1551 to different sections. For example in the field of (re) insurance data processing technology: (i) High Weight: Contract clauses, exclusions, endorsements; (ii) Low Weight: Tables, boilerplate language, notes. Further, to generate the features, for each section, the features (e.g., n-grams, semantic embeddings) are extracted that will be used to generate hashes 162 / 172.

[0048] The near duplicate detection system 1 generates, by means of a hashing unit 16, a plurality of weight vectors 161 for the plurality of datasets 141 from the unique overlapping substrings 151 by applying hashing 22 to the unique overlapping substrings 151 creating multiple weighted hashes 164 for the unique overlapping substrings 151 resulting in said weight vector 161 for each dataset 141 , wherein the plurality of weight vectors 161 result in a reduction of dimensionality of the plurality of datasets 141. As a variant, the hashing 22 is applied by using a plurality of different or alternative hash scheme 221 , as e.g. MinHash, SimHash, Sign-Random-Projections. The hashing can, however, also be applied by using only a single hash scheme 221 . For example, the near duplicate detection system 1 can e.g. processes each dataset of the large datasets 141 by cutting overlapping substrings of a fixed, predefined size 153, and hashes similar input items into same buckets 12 having a probability within the similarity threshold 1 1 by conducting min-wise independent permutations locality sensitive hashing. The unique overlapping substrings 151 can e.g. be cut into substrings of fixed size 153, the minimum size being at least 3. The hashing system 16 can e.g. introduce a sophisticated classification of document sections based on both content and structure. For instance, sections containing policy clauses, exclusions, or endorsements are classified based on their content, while tables, notes, and other non-critical elements are classified based on their structure.

[0049] Id est: the near duplicate detection system 1 assigns weights 1551 to each hash 163 generating weighted hashes 164, where the weight vectors 161 is composed of the weighted hashes 164, thus providing a weighted hashing. One of the cores of the invention lies in the assignment of different weights 1551 to the hashes 163 generated from these sections. Hashes 163 generated from semantically important sections of the datasets 141 or digital documents, such as key policy clauses or exclusions, are assigned higher weights. In contrast, hashes 163 from sections identified by the system 1 as less important, like tables or boilerplate language, receive lower weights. This inventive weighting mechanism is integrated into the hashing system 16, ensuring that the most critical parts of the digital documents 141 have a greater influence on the final similarity score, thereby improving the accuracy of the detection process. For example, when applying classification of the sections and / or hashes 163 (critical vs. non-critical) and their appropriate weight 1551 assignments in the inventive system, there may be different technical approaches which can e.g. be used depending on how the "criticality'' of a section and / or hash 163 is defined for a specific technical field. This criticality may e.g. depend on the importance or impact (which e.g. may include in the above example also financial impact strength) of that section and / or hash 163 within the context of the field-specific dataset and digital document 141 . There are a different technical approaches, the system can be realized. For example : (1 ) Static weights based on (a) definition: In this case, weights are predefined and fixed. For example, earthquake clauses might always be assigned a higher weight due to their financial impact, and / or (b) usage: Numeric values can be manually assigned based on the perceived importance of the hash 163 or section, (2) Learned weights based on: (a) definition: In this approach, the system learns which sections and / or hashes 163 (and / or sections or clauses) are more critical based on data. For instance, if a section and / or hash 163 contains clauses or sections that historically correspond to large payouts or losses, the system learns to assign a higher weight, and (b) usage: A machine learning model can be trained to predict criticality based on labeled training data; (3) rule-based weights based on: (a) definition: Weight assignment may follow a predefined set of rules based on the content of the section and / or hash 163. For example, if a section and / or hash 163 contains terms like "deductible," "coverage limit," or "threshold," it may be assigned a higher weight because these are critical financial metrics, and (b) usage: Rules may be defined like "if deductible percentage is mentioned, assign a critical weight," or "if section includes 'earthquake', assign a higher weight."

[0050] Below, an exemplary pseudo code is shown that illustrates again on a (re)insurance example, how the system can classify sections and / or hashes 163 and assign weights to them based on static, learned, or rule-based approaches. It also incorporates the example of an earthquake deductible clause. The pseudo code is given in respect hashes 163, but analogously applies to sections. phyton def classify_hash_and_assign_weight(hash_text):

[0051] # Step 1 : Identify if the hash is earthquake deductible related is_earthquake_clause = identify_earthquake_clause(hash_text)

[0052] # Step 2: Assign weight based on classification method if is_earthquake_clause: weight = assign_critical_weight(hash_text) else: weight = assign_non_critical_weight(hash_text) return weight def identify_earthquake_clause(hash_text):

[0053] # Step 1 : Look for keywords and patterns keywords = ["earthquake", "deductible", "loss threshold", "coverage limit"] for keyword in keywords: if keyword in hash_text.lower(): return True # Hash is related to earthquake deductible return False # Hash is not related to earthquake deductible def assign_critical_weight(hash_text):

[0054] # Example of static weight assignment for earthquake clauses weight = 10 # Static weight assigned to earthquake-related clauses

[0055] # Alternatively, use learned weights or rule-based system here if 'earthquake deductible' in hash_text.lower(): weight = 15 # Adjust weight based on the importance of deductible clauses

[0056] # Can add learned weights logic here (e.g., machine learning model)

[0057] # weight = learned_model.predict_criticality(hash_text) return weight def assign_non_critical_weight(hash_text):

[0058] # Example of static weight for non-critical hash (e.g., disclaimers, general terms) return 1 # Static low weight for non-critical hashes

[0059] # Example document hashes hashes = [

[0060] "The Earthquake deductible shall be 5% of the total loss, subject to a minimum of $100,000...",

[0061] "General terms and conditions apply to this agreement, including indemnity clauses...", "Losses resulting from flood or fire ore subject to specific deductibles, not applicable to earthquakes." ]

[0062] # Apply classification and weight assignment for hash in sections: weight = classify_hash_and_assign_weight(section) printff'Hash: {hash}\nAssigned Weight: {weight}\n")

[0063] In the above exemplary pseudo code the terms are given as follows (a) identify_earthquake_clause: This function scans the hash 163 for keywords like "earthquake" and "deductible." If any of these keywords appear, the hash 163 is classified as an "earthquake clause.", assign_critical_weight: This function assigns a high weight (e.g., 10 or 15) to hashes 163 related to earthquake deductibles. It is also possible to expand this with more sophisticated rules or learned weights, and assign_non_critical_weight: This function assigns a low weight (e.g., 1 ) to hashes 163 that are not critical (e.g., general terms). The types of weight assignment (static, learned, rule-based) can e.g. then be different: (1 ) Static weight assignment: In the case of earthquake-related clauses, a static weight (e.g., 10 or 15) can always be assigned, reflecting its inherent criticality. The system would e.g. apply this when there is a high level of confidence that certain clauses always have the same importance regardless of context; (2) Learned weight assignment: If the system uses historical data, the system can e.g. use machine learning structures to predict the weight dynamically based on the text. For example, if certain clauses frequently correlate with high claims, the model structure may be learned to assign a higher weight to those. The system can be trained by a classifier (e.g., a logistic regression model, or a more advanced deep learning model) using labeled data that includes both the text and the corresponding weights / criticality levels, and (3) Rule-based weight assignment: In this embodiment variant, the system uses a defined set of rules based on the presence of specific keywords or patterns. For example: If the clause mentions "deductible" and "earthquake," the system assigns a high weight (because deductible clauses are critical) . If the hash 163 talks about general terms, it assigns a low weight.

[0064] Figure 6 shows a simplified flowchart illustrating how an exemplary classification of the hashes 163 and appropriate weight assignment process in a thinned out process comprising 5 exemplary steps: Step 1 includes start processing: The system starts by processing a hash 163 of the field-specific dataset and digital document 141. In step 2, the system performs keyword checking: The system checks whether keywords or patterns related to earthquake clauses are present. In step three, the system performs classification: The section can e.g. be either classified as "earthquake" or "nonearthquake" based on the presence of these keywords. In step 3, the system assigns the generated weights. If it's an earthquake-related hash 163, the weight can be assigned dependent on the used approach (static, learned, or rule-based) . If it's a nonearthquake hash, a lower weight can e.g. be assigned. In step 5, output is generated by the system. The system assigns to each hash 163 the weight and can e.g. output the criticality (e.g., critical or non-critical) and the associated weight for each hash 163. In summary, in this embodiment variant, the technical approach of using static weighting assigns fixed numeric weights based on predefined rules, learned weighting applies a machine learning model structure to learn to assign weights dynamically to the hashes 163 based on historical data, and rule-based weighting uses rules based on keywords, clause structures, or other patterns determine the weight. The inventive technical framework allows to process and classify field-specific clauses (in the example above "earthquake deductible clauses"), while adjusting the criticality based on static rules, learned patterns, or specific clauses defined by the user.

[0065] The near duplicate detection system 1 generates, by means of a localitysensitive hashing unit 17, a plurality of band hash signature values 171 by further reducing the dimension of the weight vectors 161 applying locality-sensitive hashing to the weight vectors 161 and generating reduced weight vectors 171 comprising weighted single hashes 172 to form band hash signature values from a band of weighted vector elements of the weight vectors 161 as aggregated weighted hashes. For example, the near duplicate detection system 1 further reduces the dimension of the weight vectors 161 by applying said locality-sensitive hashing 23 creating a single hash 172 form a band of nearby elements in the weight vector 161 for generating the bandhash vector 171. Regarding the weighted hashes 164 / 172, in the present semantic-based LSH approach, the weighted hashes 164 generated from the hashing system 16 / 17 play a pivotal role in enhancing the detection of semantically similar content. Prior art LSH techniques focus on syntactic similarities, often overlooking subtle yet important semantic differences. By incorporating weighted hashes 164 / 172, the LSH process becomes much more specialized in capturing the significance of different dataset and digital document sections. As such, the inventive system 1 uses a weighted feature representation, where the known hashing process is enriched by incorporating the section weights into the feature representation. The hashing process is characterized in that (i) each token has a weight coming from the context and one coming from the section it belongs to. These two weights are multiplied, and an overall weight is computed; (ii) The probability of selecting a token during the hashing process is altered according to its weight. Tokens with higher weights are more likely to influence the hash; (iii) The resulting weighted emphasize features from more critical sections, making the hash value more sensitive to differences in these areas. This is followed by aggregating the hashes, i.e. combining the weighted hashes, where after computing the weighted hash for each feature, these are combined by the system 1 to form the final hash signature for the document and select the minimum weighted hash values across all features. Finally, the bucket assignment is applied in LSH of the locality-sensitive hashing unit 17. The weighted hashes are then used in the LSH process to place digital documents 141 into the buckets 12. The digital documents 141 with similar weighted hash signatures (i.e., those with semantically similar important sections) are more likely to fall into the same or nearby buckets. One technical important issue concerns the collision probability. However, in the inventive system 1 , the probability that two documents 141 will collide (i.e., end up in the same bucket) is higher if their semantically important sections are similar, thanks to the weighting. The processing steps can be summarized as follows: ( 1 ) Tokenize and classify sections of the document; (2) Assign weights to sections based on importance; (3) Generate weighted features and calculate weighted hashes; (4) Aggregate weighted hashes to form a document signature; (5) Use LSH to bucket documents based on weighted hash signatures.

[0066] Some of the technical advantages of the present system 1 relying on weighted hashes for the LSH are (1 ) Increased Sensitivity: The weighted hashing approach increases the sensitivity of the LSH to differences in important sections of documents 141 , reducing the likelihood of missing significant semantic variations; (2) Focused Similarity: By emphasizing critical sections, the inventive processing better aligns with the context of the technical field, e.g. (re)insurance technology, where, for example, small differences in contract clauses can lead to substantial differences in meaning and liability, and, vice-versa, large sections that are instead very similar from the point of view of text overlap, don't add to the similarity of the documents. This weighted hashing mechanism allows technically to integrate seamlessly into the LSH framework while providing the necessary emphasis on the semantic and structural importance of different document sections, making it highly effective for detecting near duplicates at scale in a reinsurance context.

[0067] The use of weighted hashes 164 / 172 ensures that sections of a digital document 141 that carry more semantic weight 1551 , such as those that directly affect sensible parts of the corresponding technological field (e.g. in (re)insurance: coverage or liability) are more strongly represented in the hash 164 / 172. This technical approach allows the LSH structure 17 to capture not just "flat" text or data similarities but also deeper, context-driven semantic connections between the documents 141. Consequently, the innovative "semantic" LSH method is particularly effective in semantic sensible technological fields and data processing, as e.g. (re)insurance industry, where small variations in used terms and language can have significant technical implications for a reliable and effective similarity detection.

[0068] As a variant, the near duplicate detection system 1 can e.g. generate a first plurality of weights 1551 for a plurality of the unique overlapping substrings 151 applying different locality-sensitive hashing schemes including at least an element of randomness. Further it can e.g. generate a second plurality of weights 161 1 for the plurality of the unique overlapping substrings 151 using the different locality-sensitive hashing schemes. It can e.g. match datasets by all pairs of weights 161 1 at least in one position, to generate candidate pairs. Finally, it can e.g. measure similarity between corresponding parts of the datasets 141 to filter for false positives generated by the automated method for the near duplicate detection system 1 .

[0069] The near duplicate detection system 1 assigns, by means of a localitysensitive hashing unit 17, datasets 141 having same or corresponding hash values 172 to the same bucket 12 of datasets 141 .

[0070] The near duplicate detection system 1 provides similarities by matching the respective unique overlapping substrings 151 and contiguous subsequences of tokens 152 and marks two datasets 141 as measured to be candidate pairs 191 1 , if for the two of respective datasets 141 the minimum number 121 for the same buckets 12 is detected to be equal or above the similarity threshold 1 1 . As a variant, the near duplicate detection system 1 can e.g. processes each dataset of the large datasets 141 by cutting overlapping substrings of a fixed, predefined size 153, and hashes similar input items into same buckets 12 having a probability within the similarity threshold 1 1 by conducting min-wise independent permutations locality sensitive hashing.

[0071] In an embodiment variant, the near duplicate detection system 1 generates a filtered set 181 of datasets 141 by removing datasets from the plurality of datasets 141 that are not detected to be similar based on the similarity threshold 1 1 . Further, it generates, by means of a filter unit 18 a plurality of partitions 183 from the filtered set of datasets resulting for each partition 183 in disjoint datasets 181 , each disjoint dataset 141 and partition 183 being assigned to a different originator unit 182. Finally, it assigns the plurality of partitions 183 to generate candidate pairs for determining dataset similarity of respective datasets 141 of the filtered set 181 of datasets to each other, within respective said partitions 183.

[0072] Accordingly, the present inventive system uses Locality Sensitive Hashing (LSH) for near duplicate detection on weighted text providing a robust solution. Important aspects improved by the invention are (i) Efficiency by processing performance and power consumption: LSH allows for fast and accurate identification of near duplicates in large datasets; (ii) Accuracy: We extend the hashes calculated via the vanilla LSH approach with an insurance-specific weight, thus making sure that the concept of similarity between documents is calculated on an industry-relevant basis; (iii) Cost Reduction: Reducing redundant data lowers storage and processing costs; (iv) Data Integrity: Improved accuracy and consistency in records enhance overall data integrity; and (v) Regulatory Compliance: Better data management practices support compliance with industry regulations.

[0073] In the inventive system, the LSH structure with the appropriate functions are specifically designed so that hash value collisions are more likely for two input values that are close together than for inputs that are far apart. Id est, the disclosed system provides an efficient and robust operation in approximate nearest neighbor (ANN) search. The term "close together" means that the data points that are within a certain distance bounded by a threshold value. In other words, in the present system, if two points are close together, then after a "projection" operation these two points will remain close together. As figure 1 shows, for example, the system can be realized with k =4, m =4, L =6: sharing of the first-level partition among the different hash functions. Hash tables (ul , u2), (u 1 , u3) and (ul , u4) are first partitioned according to ul . Each partition (shown in different colors) is then partitioned according to hash functions u2, u3 and u4. The corresponding hash values share the results of the first-level partitions among the different hash functions. For instance, consider the hash tables for gi,2= (ui, U2) and gi,3 = (ui, U3). Both these tables are partitioned first using the same hash function ul . Instead of repeating this partitioning work, instead the first level partitioned table among these hashes can be shared. Figure 1 shows an example of this sharing. In order to achieve this, a 2-step process can be used: the data are portioned on the basis of the m hash functions ui...um. This step only involves m partitions. In the second step, each first level partition are taken, say Ui, and partition it on the basis of Ui+i...um. This involves a total of L = m / 2 partitions. This reduces the number of partitioning steps from 2 * L (first and second level partitions) to L+m, which takes much less time since m ~ O( L) .

[0074] It is an advantage of the present invention, that it provides a system allowing to compare efficiently millions or even billions of samples, and all of them.

[0075] With prior art systems, even on the best hardware, comparing all pairs is out of the question. This produces an at best complexity of O(n2). Even if comparing a single query against the billions of samples, an at best complexity of O(n) id achieved.

[0076] Technically, form the point of runtime efficiency, also the complexity behind a single similarity calculation has to be considered (every sample needs to be stored as a vector, often very highly-dimensional vectors) this increases our complexity even further. The present inventive system allows to avoid such operational technical problems in data processing, allowing to perform a search with sub-linear complexity using the proposed approximate search. Rather than comparing every vector (providing an exhaustive search, the present system approximate and limit the search scope to only the most relevant vectors. Thus, LSH structure, as disclosed herein, provides the system with such sub-linear search times.

[0077] In the present system, the complexity of finding similar pairs of vectors is considered, where the number of calculations required to compare everything is found to be unmanageably enormous even with reasonably small datasets. Starting with a vector index, If just one new vector is introduced and attempted to find the closest match, vector to every other vector is to be compared in the database. This generates 19 a linear time complexity, which cannot scale to fast search in larger datasets. The technical problem becomes even worse if the system is to compare all of those vectors against each other. The optimal approach sorting structure to achieve this is at best log-linear time complexity. So, a technical way is required to reduce the number of comparisons. Ideally, only vectors should be compared that are believed to be potential matches or candidate pairs. The herein disclosed Locality sensitive hashing (LSH) structure allows to do this. At its core, the final LSH function allows to segment and hash the same sample several times. And when the system detects that a pair of vectors has been hashed to the same value at least once, the system tags them as candidate pairs, that is, potential matches. Thus, the system derives a key-value pair which is fed into a dictionary. The key is processed through a dictionary hash function and mapped to a specific bucket. The system then connects the respective value to this bucket. However, there is a technical key difference between those types of hash functions, as used in prior art systems and that used in the herein proposed LSH structure. With dictionaries, the goal is to minimize the chances of multiple key-values being mapped to the same bucket, i.e. the goal is to minimize collisions. With the herein proposed LSH structure, the technical goal is almost the opposite. In the present LSH structure, the system wants to maximize collisions, although ideally only for similar inputs. It is to be noted that the present system provides no single approach to hashing in LSH. Indeed, they all share the same ‘bucket similar samples through a hash function' logic, but they can vary beyond this. This approach makes the method unique. Figure 3 shows a diagram illustrating an exemplary LSH structure according to the invention aiming to place similar values into the same buckets.

[0078] As an embodiment variant, the system can be realized to control a determination or measurement of document similarity by nodes of a plurality of computing devices. In one example, dimensionality of the documents that are stored electronically (e.g., digital documents (structured or unstructured)) is reduced by a computing device through computation of wight data from the documents, e.g., via a hashing technique such as "minhashing" which is also known as min-wise independent permutations locality sensitive hashing. This works to reduce the dimensionality of the dataset (i.e., randomness of variables in the dataset) formed by the documents while retaining information used to calculate similarity between underlying documents by the computing device. From this weight data, locality sensitive hashing structures can be used by the computing device to determine similarity of the weight data to each other, such as by hashing values of the weight data into respective buckets (e.g., ranges, collections, or clusters of similar hashes) . Thus, assignment of the weight data into respective buckets may be used to determine similarity of the weight data and thus corresponding documents to each other. In one example, this is done by the computing device through use of threshold data describing a minimum number of buckets that are to be shared by respective weight data in order to be considered similar.

[0079] Finally, the data processing workflow can e.g. comprise a final step of postfiltering. Thus, the data processing workflow is given by the step of tokenizing — ► classifying — ► weighting — ► hashing — ► bucketing — ► post-filtering. The "post-filter" step may be realized to include a human-in-the-loop review allowing to generate a confidence score parameter value. Thus, the present invention provides a complete, autonomously operating end-to-end system.

[0080] In an embodiment variant, the weighted hash generation and bucket search can be parallelized across nodes in a distributed system to technically improve speed and scalability, especially when dealing with millions of documents or clause templates, as it is characteristics and often a technical precondition for data processing in the field of reinsurance technology, but also in other technological fields. As an embodiment example, to match extracted clauses to a known library (or normalize variants), you might use weighted hashing (like SimHash, MinHash, or custom TF-IDF + hash embedding) and bucket search. The hashing can then be realized by then first generate a weighted fingerprint (SimHash or MinHash) of each clause (considering term weights from TF-IDF or attention weights from transformer embeddings), and then use a locality-sensitive hashing (LSH) structure to map similar clauses into the same bucket, as described in detail above. For the technical realization of the parallelization, the field-specific datasets and digital documents 141 can then be distributed across nodes (horizontal scaling) . Each node parses and normalizes datasets and digital documents 141 , generates hashes 163 and weights (as described above), and assigns to each hash 163 the weight. Finally, each node conducts the assignment to the LSH buckets (as described above) . In an embodiment variant the later assignment can at least partially or completely be to an ANN index). For a bucket search, in the specific embodiment variant, when querying (e.g., for clause normalization or duplicate detection), the system computes the clause hash 163. The system may only query the corresponding LSH buckets for similar hashes 163. The system may then also use e.g. approximate nearest neighbor (ANN) search (e.g., with FAISS or Annoy) for efficient retrieval. As a technical frameworks for the parallelization, the system may e.g. be realized using Apache Spark, Ray, or Dask to coordinate distributed execution. For storage / indexing the system may e.g. use Redis, FAISS, or ElasticSearch (for vector and hash-based search). The disclosed parallelization structure has various technical advantages as e.g. providing improved scalability. For example, it allows to handle the system large-scale datasets and digital documents 141 , as e.g. large-scale reinsurance portfolios and treaty libraries. Further, it allows the system to reduce latency providing a faster matching and normalization during ingestion or querying. Finally, it results in better overall resilience of the system by distributing fault-tolerance in clause parsing pipelines. Thus, by the technical structure of the present system, the weighted hash generation and bucket search can (and for specific application should) be parallelized, for example using the above described LSH technique (which can in an embodiment variant also be combined with supporting ANN techniques) distributed across a plurality of data processing nodes. The known prior art systems are typically limited by the technical structure on single-machine LSH. Again, the system 1 allows distributing the weighted-hash index across multiple nodes or cloud functions: The ease of parallelization of this approach is part of the novelty and also of the technical effect achieved by the system 1 .

[0081] Finally, in an embodiment variant, privacy and anonymization can e.g. be realized in the above discussed parsing and hashing pipeline. In particular, using the present system redacting personal data before hashing is not only possible to implement, but may be recommended when processing datasets and digital documents 141 of sensitive fields as e.g. sensitive insurance or reinsurance documents, health-related datasets and digital documents 141 etc.. This falls under privacypreserving processing, and the technical strategies for integrating anonymization or redaction before any parsing, hashing, or storage in the present system will be discussed below. The privacy and anonymization structures can e.g. be integrated in the system before hashing or clause normalization begins, i.e. during preprocessing (after OCR and tokenization), and / or optionally, during / after NLP entity recognition (NER). To detect PH or sensitive data in the datasets and digital documents 141 , the system can e.g. comprise a PH (Personally Identifiable Information) detection module based on Named Entity Recognition (NER) e.g. to identify names and / or addresses and / or mail, phone numbers and / or policy numbers, and / or client-specific values, and to regexe for structured data (e.g. phone numbers, IDs). To regexe for structured data means here using regular expressions (regexes) to detect and extract structured data elements embedded within unstructured or semi-structured text. PH, as used herein, is a textual reference to personal data that could be used to identify an individual. Thus, as discussed above, PH examples can include addresses, bank account numbers, and phone numbers. With PH detection, the system not only locate the Pll entities but also redact the Pll entities in the text. The Pll detection module can e.g. using systems as SpaCy with custom NER, Presidio (Microsoft), AWS Comprehend (Pll mode) and / or Google's DLP API etc.. Subsequent to Pll detection, the Pll can e.g. be redacted or replaced before hashing. For example, the system apply reduction by replacing sensitive tokens with [REDACTED] or the system replaces Pll with token, e.g. <NAME>, <ADDRESS>, <POLICY_ID>. As an embodiment variant the system can e.g. also drop Tokens removing them completely if semantically irrelevant. For example, an original clause may be "The Earthquake deductible applies to John Smith's building located at 1234 Market Street and is 5% of TIV." After system's Pll redaction, the clause may look like 'The Earthquake deductible applies to <NAME>'s building located at <ADDRESS> and is 5% of TIV." After applying the Pll redaction, the system can continue with the above described clause vectorization and hashing. This technically ensures that semantically similar clauses still match, without leaking personal information. Further, another technical advantage is that regarding semantic integrity, the disclosed redaction personal data does not harm clause normalization, because (i) Key structural and semantic elements (like "Earthquake deductible", "% of TIV", "minimum") remain intact, and (ii) Models trained to recognize clause intent are generally robust to name / address tokens. However, for specific sensitivity dependencies, the system needs additional tuning to capture those cases, as for example, when the deductible amount itself is tied to identifiable entity data (e.g. insured value unique to a client). You might need to generalize or pseudonymize numeric values too. Further privacy and compliance issues can easily be integrated in the technical structure of this embodiment variant, if required, which is a further technical advantage over the known prior art systems. For example, to achieve (i) GDPR / CCPA compliance, the anonymization can be applied before storage or hash generation, (ii) audit trail, Teh system can e.g. keep logs of redaction applied, not the original content, (iii) data minimization, the system can e.g. only retain normalized clause schema and drop raw text, and (iv) improved secure hashing, the system can use non-reversible hashes if storing hashed versions. As these embodiment examples show, the inventive structure of the system allows technically an improved adaptability.

[0082] Further, the inventive application of Locality-Sensitive Hashing (LSH) is used for the approximate similarity search in high-dimensional spaces (clause vectors), is not the only technical option. Depending on or triggered by requirements given by characteristics of the large datasets 141 / 5ii e.g. comprising processing speed, and / or accuracy, privacy, and / or data volume, the applied structure can be selected out of applying locality-sensitive hashing or Approximate Nearest Neighbor (ANN) libraries searching or transformer embedding together with cosine similarity or knowledge graph with ontology matching or inverted index with token fingerprinting or fuzzy string matching with edit distance structures. Each of the above mentioned structures has their own technical advantage dependent on the specification or characteristics of the large datasets 141 / 5ii:

[0083] (A) Approximate Nearest Neighbor (ANN) Libraries: Optimized vector similarity search libraries can be applied to replace or outperform LSH for some of specific real-world use cases the large datasets 141 / 5ii. Examples are (i) FAISS (Facebook Al Similarity Search) which can be highly optimized for CPU / GPU, and allows for multiple indexing options (IVF, HNSW, PQ); (ii) Annoy (Spotify), which can be good for memory-constrained environments and fast tree-based indexing; (iii) ScaNN (Google) which is shows often a state-of-the-art vector search performance; (iv) NMSLIB (Non-Metric Space Library), which allows to supports HNSW, SW-graph, and other fast ANN algorithms. Optimized vector similarity search libraries can show technical advantages of applying LSH for the present inventive system, in case the system has to work with clause embeddings (e.g., BERT vectors), and / or needs faster, more scalable similarity search than LSH, and / or technically requires to be enabled for or comprise pre-index clause templates for fast matching.

[0084] (B) Transformer embedding and cosine similarity: Instead of hashing, the system can e.g. use sentence embeddings (e.g., from BERT, SBERT) and cosine similarity. In the present system, for example, the system can e.g. in a first step encode clauses using a transformer (like sentence-transformers). In a second step normalize the vectors, as already described and store them, in a third step, in a vector database (as e.g., Pinecone, Weaviate, Milvus, or FAISS etc.) . Finally, the system can then in a last step be configured to use cosing similarity to retrieve closest matches. This embodiment variant has, inter alia, the advantage that it allows to capture semantics (not just surface similarity) if required by the clause similarity. Further, it has the advantage that it allows the system to generate an interpretable ranking by similarity score. In particular, it has efficiency and performance advantages, when the system uses pre-trained or finetuned language models.

[0085] (C) Knowledge Graph and Ontology Matching: For highly structured data (in the above example of the (re)insurance field, for example, (re)insurance clauses, the system can be realized to use a semantic model structure instead of fuzzy hashing. In this embodiment variant, the system generates an ontology automatically from the large datasets 141 / 5ii. In the above example, the system e.g. creates an ontology of reinsurance clause types, perils, deductible structures. To realize this component, the system comprises, in this embodiment variant, a clause parser which maps text to ontology terms (e.g. using rule-based and machine learning (ML) structures. Further, the system applies, i.e. is configured to apply a graph matching structure, not hashing, to normalize or compare clauses, which is an important technical difference. The technical advantages of this embodiment variant are that it provides a particular fieldspecific, i.e. a domain-aware, data processing. For example, if the system is enabled to capture domain-specific knowledge (e.g., "5% of TIV" is always mapped to a known node). Such domain-specific knowledge can e.g. be captured automatically e.g. by machine learning structures, or directly implemented. Another advantage is that it provides an interpretable and rule-auditable matching.

[0086] (D) Inverted Index and Token Fingerprinting: As an embodiment variant for smaller clause libraries or highly templated clauses, also automated information retrieval (IR) structures can e.g. be used for the system. The automated information retrieval (IR) structure first tokenizes the large datasets 141 / 5ii. In a next step, the large datasets 141 / 5ii then can be indexed, e.g., BM25 or TF-IDF weighted inverted index (e.g., via ElasticSearch or Whoosh) . Finally, the system then matches new clauses using similarity scoring. This embodiment variant is particularly advantageously, if the system is applied to process documents, which are textual and e.g. contain repeated boilerplate, where boilerplate-heavy documents are documents that contain a significant amount of standardized, reusable text (referred to as boilerplate language), with relatively few changes across different instances. Further, this embodiment variant has the advantage that it provides an interpretable search process with traditional IR techniques. In particular, it has the advantage that in this embodiment variant, if the dataset size of the large datasets 141 / 5ii is moderate, it provides an efficient data processing reducing the required processing power.

[0087] (E) Fuzzy String Matching / Edit Distance: In this embodiment variant, the system comprises one or more character-level or token-level edit distance structures for smaller or simpler datasets 141 / 5ii. For example, the character-level or token-level edit distance structures can e.g. comprise Levenshtein distance, or Jaccard similarity on token sets, or FuzzyWuzzy (string scoring wrapper), which showed to be particular suitable to process smaller or simpler datasets 141 / 5ii with the present inventive system. This embodiment variant can e.g. be advantageously, if the datasets 141 / 5ii comprises clauses that are short, and variations are limited, if the computational costs are acceptable (in this embodiment variant, they may not scale well), and if no embeddings or ML implementation is possible. Thus, it is a technical advantage that this embodiment variant can be realized without requiring the implementation of ML structures.

[0088] As an embodiment variant, the system 1 can e.g. further comprises a collision threshold parameter, which is dynamically adapted to possible segmentations in the large datasets and / or digital documents 141 tuning the similarity and / or matching sensitivity of the system 1 in the hashing or clause matching of the system based on the segmentation. Tuning the similarity or match sensitivity in the hashing and / or clause matching can e.g. be based on the type of business covered by the large datasets and / or digital documents 141 . In particular, in the above example of datasets in the field of reinsurance, the collision threshold can e.g. be dynamically adapted to treaty LoB (Line of Business), which refers herein to tuning the similarity or match sensitivity in the hashing or clause matching based on the type of business the treaty covers, for example, Property, Casualty, Marine, Life, etc.

[0089] The collision threshold in the present near duplicate detection system 1 and the present context of applying Locality-Sensitive Hashing (LSH) or other similarity-based matching, a "collision" means two clauses end up in the same bucket, i.e., they're considered to be similar by the system. A collision threshold defines how similar two items must be to fall into the same bucket or to be considered a match. Id est, a high threshold signals more lenient, i.e. more collisions, resulting in potentially lower precision. A low threshold signals stricter, i.e. fewer collisions, resulting in potentially higher precision. In this embodiment variant, instead of using a fixed threshold for all treaty clauses or business lines, the system 1 dynamically adjusts the threshold depending on the Line of Business (LoB). The system 1 can be required to adapt by Treaty LoB, since different lines of business have different clause structures, terminology, and variability. In the above example of the reinsurance field, it can be illustrated exemplary by the following table:

[0090] For example, in the sector of property, clauses are often more standardized, so a stricter threshold may work well to avoid false positives. In contrast, in Marine or Casualty, clauses can vary widely in structure and wording, so a looser threshold may be needed to catch legitimate matches. In the present invention, the system 1 comprises a clause normalization with LSH (or vector-based) similarity matching. When a new clause is ingested by the system 1 , the system 1 can e.g.: in a step 1 detect the treaty's LoB from metadata or classification and in a step 2 use and / or apply the similarity threshold to consider clauses as matching (e.g. for Property treaties, only clauses can be considered by the system 1 as matches if similarity > 0.90, while for Casualty, the threshold is lowered by the system 1 to 0.75, and for Marine, the system can even allow down to 0.70, etc. This has the advantage to reduce false positives in standardized LoBs, and increase recall in complex, less standardized LoBs. In the inventive clause normalization, especially in the example for automating reinsurance treaty ingestion by the system 1, overmatching can cause wrong clause interpretations (e.g., matching a wind deductible with an earthquake one), whereas undermatching leads to failing to recognize semantically identical clauses with varied wording — especially common in Casualty and Marine. This embodiment variant provides dynamic adaptation which balance this LoB-specific tradeoff. Thus, collision threshold dynamically adapted to treaty LoB, which is means that the similarity or match sensitivity of the system 1 used in hashing / matching clauses is dynamically adjusted based on the specific Line of Business of the treaty improving accuracy by accounting for how clause variation differs across business types.

[0091] Finally, it should be noted again that the present inventive system explicitly enables downstream data processing and analytics. For example, bucket assignments can be logged and reused for aggregate reporting (e.g., in the example of the reinsurance field, treaty-portfolio homogeneity detection and analysis).

[0092] List of References

[0093] 1 Near duplicate detection system

[0094] 1 1 Similarity threshold value

[0095] 12 Buckets

[0096] 121 Minimum number of buckets (given by the similarity threshold value)

[0097] 13 Interface

[0098] 14 Data storage, persistence storage

[0099] 141 Datasets and / or digital documents

[0100] 15 Tokenizer (w-shingling)

[0101] 151 Unique overlapping substrings

[0102] 152 Contiguous subsequences of tokens

[0103] 153 Fixed predefined size w

[0104] 154 Data transfer: unit 15 to unit 16

[0105] 155 Weighting unit

[0106] 1551 Technology field specific weights

[0107] 1552 Custom pre-processing

[0108] 1553 Stemming process

[0109] 1554 Structure detection

[0110] 16 Hashing unit

[0111] 161 Weight vectors

[0112] 161 1 Weights or signatures included in the weight vector

[0113] 162 Data transfer: unit 16 to unit 17

[0114] 163 Hashes

[0115] 164 Weighted hashes

[0116] 17 Locality-sensitive hashing unit

[0117] 171 Reduced weight vectors / Band hash signature vectors 172 Hash parameter value

[0118] 173 Data transfer: unit 17 to trigger 19

[0119] 174 Comparator

[0120] 18 Filter unit

[0121] 181 Disjoint filtered set of datasets and / or digital documents

[0122] 182 Originator unit / Creator unit

[0123] 183 Partition

[0124] 19 Detection unit

[0125] 191 Data storage

[0126] 191 1 Candidate pairs of datasets

[0127] 192 Comparator (assignment)

[0128] 193 Trigger

[0129] 194 Purged datasets

[0130] 194i Purged dataset of industrial field unit 5i Processing steps

[0131] 21 Cutting / w-shingling

[0132] 22 Hashing

[0133] 22i Different hash scheme

[0134] 23 Locality-sensitive hashing

[0135] 24 Data purging Data transfer network Data purge engine Technologic field units

[0136] 51 Industrial plants

[0137] 51 i Datasets related to industrial plants

[0138] 52 Monitoring systems

[0139] 52i Datasets related to monitoring or smart systems 53 Risk-transfer systems

[0140] 53i Datasets related to risk-transfer systems

Claims

Claims1. An automated method for a near duplicate detection system (1 ) for efficient similarity detection in large datasets and / or digital documents (141 ), comprising the steps of: receiving an input specifying a similarity threshold (1 1 ) via an interface (13), the similarity threshold (1 1 ) defining a minimum number (121 ) of a plurality of buckets (12) where two of respective datasets (141 ) are measured to be similar, if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ); assessing a data storage (14) comprising the large datasets (141 / 511) of one or a plurality of field-specific units (5 / 51) and cutting (21 ) each dataset (141 ) in unique overlapping substrings (151 ) each of which being composed of contiguous subsequences of tokens (152) within the respective dataset (141 ); generating a plurality of weight vectors (161 ) for the plurality of datasets (141 ) from the unique overlapping substrings (151 ) by applying hashing (22) to the unique overlapping substrings (151 ) creating multiple hashes (163) for the unique overlapping substrings (151 ), and weighting the generated hashes (163) by assigning weights to the hashes ( 163) and providing weighted hashes ( 164) resulting in said weight vector (161 ) for each dataset (141 ), wherein the plurality of weight vectors (161 ) result in a reduction of dimensionality of the plurality of datasets (141 ); generating a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying locality-sensitive hashing to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes ( 172) to form band hash signature values from a band of weighted vector elements of the weight vectors (161 ) as aggregated weighted hashes; assigning datasets ( 15) having same or similar hash values ( 172) to the same bucket (12) of datasets (15); andproviding similarities by matching the respective unique overlapping substrings (151 ) and contiguous subsequences of tokens (152) and marking two datasets (141 ) as measured to be candidate pairs (191 1 ), if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ) .

2. An automated method for a near duplicate detection system (1 ) according to claim 1 , wherein for generating a plurality of weight vectors (161 ) and weighting the generated hashes (163) by assigning weights to the hashes (163), the system (1 ) comprises a supervised machine learning structure applied to the large datasets and / or digital documents (141 ), the weights being learned by said supervised machine learning structure on the large datasets and / or digital documents (141 ).

3. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 or 2, wherein the weight vectors (161 ) are normalized by the system (1 ) so that the condition= 1 is met.

4. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 3, wherein the system (1 ) comprises a collision threshold parameter, which is dynamically adapted to possible segmentations in the large datasets and / or digital documents (141 ) tuning the similarity and / or matching sensitivity of the system (1 ) in the hashing or clause matching of the system based on the segmentation.

5. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 4, wherein tuning the similarity or match sensitivity in the hashing and / or clause matching is based on the type of business covered by the large datasets and / or digital documents (141 ) .

6. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 5, further comprising the steps of: generating a first plurality of weights (161 1 ) for a plurality of the unique overlapping substrings (151 ) applying different locality-sensitive hashing schemes including at least an element of randomness;generating a second plurality of weights (1611 ) for the plurality of the unique overlapping substrings (151) using the different locality-sensitive hashing schemes; matching datasets by all pairs of weights (161 1 ) at least in one position, to generate candidate pairs; and measuring similarity between corresponding parts of the datasets (141 ) to filter for false positives generated by the automated method for the near duplicate detection system (1 ) .

7. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 6, further comprising the steps of: generating a filtered set (181 ) of datasets (141 ) by removing datasets from the plurality of datasets (141 ) that are not detected to be similar based on the similarity threshold (1 1 ); generating, by means of a filter unit (18) a plurality of partitions (183) from the filtered set of datasets resulting for each partition (183) in disjoint datasets (181 ), each disjoint dataset (141 ) and partition (183) being assigned to a different originator unit (182); and assigning the plurality of partitions (183) to generate candidate pairs for determining dataset similarity of respective datasets (141 ) of the filtered set (181 ) of datasets to each other, within respective said partitions (183) .

8. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 7, further comprising the steps of: processing each dataset of the large datasets (141 ) by cutting overlapping substrings of a fixed, predefined size (153);hashing similar input items into same buckets (12) having a probability within the similarity threshold (1 1 ) by conducting min-wise independent permutations locality sensitive hashing.

9. An automated method for a near duplicate detection system (1 ) according to claim 8, wherein the hashing is applied by using a plurality of different hash scheme (221 ) .

10. An automated method for a near duplicate detection system (1 ) according to claim 8, wherein the hashing is applied by using only a single hash scheme (221 ) .1 1. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 10, wherein hashes 163 and sections are weighted.

12. An automated method for a near duplicate detection system (1 ) according to claim 1 1 , wherein for a bucket assignment, a collision probability factor tuned by the system based on a weight distribution by adjusting collision probability using a monotonic relation of cumulative section-weights.

13. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 12, further comprising the steps of: further reducing the dimension of the weight vectors (161 ) by applying said locality-sensitive hashing (23) creating a single hash (172) form a band of nearby elements in the weight vector (161 ) for generating the bandhash vector (171 ) .

14. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 13, wherein the unique overlapping substrings (151 ) is cut into substrings of fixed size (153), the minimum size being at least 3.

15. An automated method for a near duplicate detection system (1 ) according to one of the claims 1 to 14, wherein the large datasets and / or digital documents (141 ) comprise reinsurance corpora.

16. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents, the detection system being configured to: receive an input specifying a similarity threshold (1 1 ) via an interface (13), the similarity threshold (1 1 ) defining a minimum number (121 ) of a plurality of buckets (12) where two of respective datasets (141 ) are measured to be similar, if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ); assess a data storage (14) of the system (1 ) comprising the large datasets (141 ) stored by means of data structures of the data storage (14) and, by means of a tokenizer (15), to cut (21 ) each dataset (141 ) in unique overlapping substrings (151 ) each of which being composed of contiguous subsequences of tokens (152) within the respective dataset (141 ); generate a plurality of weight vectors (161 ) for the plurality of datasets (141 ) from the unique overlapping substrings (151 ) by applying hashing (22) to the unique overlapping substrings (151 ) creating multiple hashes (163) for the unique overlapping substrings (151 ), and weight the generated hashes (163) by assigning weights to the hashes (163) and providing weighted hashes ( 164) resulting in said weight vector (161 ) for each dataset (141 ), wherein the plurality of weight vectors (161 ) result in a reduction of dimensionality of the plurality of datasets (141 ); generate, by means of a locality-sensitive hashing unit (17), a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying locality-sensitive hashing to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes (172) to form band hash signature values from a band of weighted vector elements of the weight vectors (161 ) as aggregated weighted hashes; assign, by means of a locality-sensitive hashing unit (17), datasets (141 ) having same or corresponding hash values ( 172) to the same bucket (12) of datasets (141 ); andprovide similarities by matching the respective unique overlapping substrings (151 ) and contiguous subsequences of tokens (152) and mark two datasets (141 ) as measured to be candidate pairs (191 1 ), if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ).

17. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to claim 16, further being configured to: generate a first plurality of weights (161 1 ) for a plurality of the unique overlapping substrings (151 ) applying different locality-sensitive hashing schemes including at least an element of randomness; generate a second plurality of weights (161 1 ) for the plurality of the unique overlapping substrings (151 ) using the different locality-sensitive hashing schemes; match datasets by all pairs of weights (161 1 ) at least in one position, to generate candidate pairs; and measure similarity between corresponding parts of the datasets (141 ) to filter for false positives generated by the automated method for the near duplicate detection system (1 ) .

18. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to one of the claims 16 or 17, further being configured to: process each dataset of the large datasets (141 ) by cutting overlapping substrings of a fixed, predefined size (153); hash similar input items into same buckets (12) having a probability within the similarity threshold (1 1 ) by conducting min-wise independent permutations locality sensitive hashing.

19. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to claim 18, wherein for the hashing is applied by using a plurality of different hash scheme (221 ).20 A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to claim 1 , wherein for the hashing is applied by using only a single hash scheme (221 ) .

21. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to one of the claims 16 to 20, further being configured to: further reduce by applying said locality-sensitive hashing (23) by creating a single hash ( 172) form a band of nearby elements in the weight vector (161 ) by generating a bandhash vector (171 ) .

22. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents according to one of the claims 16 to 21 , wherein the unique overlapping substrings (151 ) is cut into substrings of fixed size (153), the minimum size being at least 3.

23. An automated method for a near duplicate detection system (1 ) for efficient similarity detection in large datasets and / or digital documents (141 ), comprising the steps of: receiving an input specifying a similarity threshold (1 1 ) via an interface (13), the similarity threshold (1 1 ) defining a minimum number (121 ) of a plurality of buckets (12) where two of respective datasets (141 ) are measured to be similar, if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ); assessing a data storage (14) comprising the large datasets (141 / 5ii) of one or a plurality of field-specific units (5 / 5i) and cutting (21 ) each dataset (141 ) in unique overlapping substrings (151 ) each of which being composed of contiguous subsequences of tokens (152) within the respective dataset (141 );generating a plurality of weight vectors (161 ) for the plurality of datasets (141 ) from the unique overlapping substrings (151 ) by applying hashing (22) to the unique overlapping substrings (151 ) creating multiple hashes (163) for the unique overlapping substrings (151 ), and weighting the generated hashes (163) by assigning weights to the hashes ( 163) and providing weighted hashes ( 164) resulting in said weight vector (161 ) for each dataset (141 ), wherein the plurality of weight vectors (161 ) result in a reduction of dimensionality of the plurality of datasets (141 ); generating a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying a similarity search structure for the high-dimensional space of the large datasets (141 / 5ii) to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes ( 172) to form band hash signature values from a band of weighted vector elements of the weight vectors (161 ) as aggregated weighted hashes; assigning datasets ( 15) having same or similar hash values ( 172) to the same bucket (12) of datasets (15); and providing similarities by matching the respective unique overlapping substrings (151 ) and contiguous subsequences of tokens (152) and marking two datasets (141 ) as measured to be candidate pairs (191 1 ), if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ) .

24. An automated method for a near duplicate detection system (1 ) according to claim 23, further comprising the step of: generating a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying as similarity search structure locality-sensitive hashing or Approximate Nearest Neighbor (ANN) libraries searching or transformer embedding together with cosine similarity to retrieve closest matches or knowledge graph with ontology matching or inverted index with token fingerprinting or fuzzy string matching with edit distance structures to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes ( 172) to form band hash signature values from a band of weighted vector elements ofthe weight vectors (161 ) as aggregated weighted hashes, wherein the appropriate search structure is selected depending on or triggered by requirements given by characteristics of the large datasets (141 / 5ii) comprising processing speed, and / or accuracy, privacy, and / or data volume.

25. A near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents (141 ), the detection system (1 ) being configured to: receive an input specifying a similarity threshold (1 1 ) via an interface (13), the similarity threshold (1 1 ) defining a minimum number (121 ) of a plurality of buckets (12) where two of respective datasets (141 ) are measured to be similar, if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ); assess a data storage (14) of the system (1 ) comprising the large datasets (141 ) stored by means of data structures of the data storage (14) and, by means of a tokenizer (15), to cut (21 ) each dataset (141 ) in unique overlapping substrings (151 ) each of which being composed of contiguous subsequences of tokens (152) within the respective dataset (141 ); generate a plurality of weight vectors (161 ) for the plurality of datasets (141 ) from the unique overlapping substrings (151 ) by applying hashing (22) to the unique overlapping substrings (151 ) creating multiple hashes (163) for the unique overlapping substrings (151 ), and weight the generated hashes (163) by assigning weights to the hashes (163) and providing weighted hashes ( 164) resulting in said weight vector (161 ) for each dataset (141 ), wherein the plurality of weight vectors (161 ) result in a reduction of dimensionality of the plurality of datasets (141 ); generate, by means of a locality-sensitive hashing unit (17), a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying a similarity search structure for the high-dimensional space of the large datasets (141 / 5ii) to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes (172) to form band hash signature values from a band of weighted vector elements of the weight vectors (161 ) as aggregated weighted hashes;50 assign, by means of a locality-sensitive hashing unit (17), datasets (141 ) having same or corresponding hash values ( 172) to the same bucket (12) of datasets (141 ); and provide similarities by matching the respective unique overlapping substrings (151 ) and contiguous subsequences of tokens (152) and mark two datasets (141 ) as measured to be candidate pairs (191 1 ), if for the two of respective datasets (141 ) the minimum number (121 ) for the same buckets (12) is detected to be equal or above the similarity threshold (1 1 ).

26. The near duplicate detection system ( 1 ) for efficient similarity detection in large datasets of documents (141 ) according to claim 23, the detection system (1 ) further being configured to: generate a plurality of band hash signature values (171 ) by further reducing the dimension of the weight vectors (161 ) applying as similarity search structure localitysensitive hashing or Approximate Nearest Neighbor (ANN) libraries searching or transformer embedding together with cosine similarity to retrieve closest matches or knowledge graph with ontology matching or inverted index with token fingerprinting or fuzzy string matching with edit distance structures to the weight vectors (161 ) and generating reduced weight vectors (171 ) comprising weighted single hashes ( 172) to form band hash signature values from a band of weighted vector elements of the weight vectors (161 ) as aggregated weighted hashes, wherein the appropriate search structure is selected depending on or triggered by requirements given by characteristics of the large datasets (141 / 5ii) comprising processing speed, and / or accuracy, privacy, and / or data volume.

27. The near duplicate detection system ( 1 ) for efficient similarity detection in large datasets of documents (141 ) according to one of the claims 25 or 26, the detection system (1 ), for generating a plurality of weight vectors (161 ) and weighting the generated hashes (163) by assigning weights to the hashes (163), comprises a supervised machine learning structure applied to the large datasets and / or digital documents (141 ), the weights being learned by said supervised machine learning structure on the large datasets and / or digital documents (141 ) .

28. The near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents (141 ) according to one of the claims 25 to 27, wherein the weight vectors (161 ) are normalized by the system (1 ) so that the condition Dr= 1 is met.

29. The near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents (141 ) according to one of the claims 25 to 28,, wherein the system (1 ) comprises a collision threshold parameter, which is dynamically adapted to possible segmentations in the large datasets and / or digital documents (141 ) tuning the similarity and / or matching sensitivity of the system (1 ) in the hashing or clause matching of the system based on the segmentation.

30. The near duplicate detection system (1 ) for efficient similarity detection in large datasets of documents (141 ) according to one of the claims 25 to 29, wherein tuning the similarity or match sensitivity in the hashing and / or clause matching is based on the type of business covered by the large datasets and / or digital documents (141 ) .

Images