Cross-wheel and cross-piece de-overlapping combination method and device and electronic equipment

Abstract

The invention provides an inter-wheel and inter-slice de-overlapping merging method and device and electronic equipment, and relates to the field of data processing.The method comprises the steps that the same file to be processed and the corresponding same extraction problem are obtained, and the file and the extraction problem serve as basic task carriers; performing multi-round and / or multi-piece extraction operation of the target information on the basic task carrier to obtain extracted candidate data packets; establishing an interval placeholder corresponding to the file and the extraction problem, and verifying whether a new candidate interval corresponding to a hit interval coordinate in the candidate data packet is overlapped with an existing effective interval recorded in the interval placeholder to obtain a verification result; if the verification result is that overlapping does not exist, the coordinates of the hit interval are written into an interval placeholder, cross-wheel and cross-piece interval placeholder locking is completed, new candidate objects are determined as effective candidate objects, context windows of the effective candidate objects are synchronized, and cross-wheel and cross-piece de-overlapping and merging processing is carried out based on data of all the effective candidate objects.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, and electronic device for cross-wheel and cross-piece deduplication and merging. Background Technology

[0002] Currently, in the field of natural language processing, information extraction is a core technology for applications such as intelligent question answering, knowledge graph construction, and text analysis. The accuracy and stability of its results directly affect the performance of downstream tasks. With the expansion of data scale and the increasing refinement of extraction requirements, multi-round extraction and multi-segment extraction (referred to as multi-round / multi-segment extraction) have become the mainstream modes for processing complex text information. Through multiple iterations or multi-dimensional segment retrieval, comprehensive coverage of target information can be achieved. However, existing multi-round / multi-segment extraction results contain a large number of duplicate and conflicting entries, leading to low reliability of the information extraction results. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, and electronic device for deduplication and merging across wheels and slices, in order to solve the technical problem of low reliability of information extraction results.

[0004] Firstly, this application provides a method for deduplication and merging across batches and slices, the method comprising: Obtain the same file to be processed and the corresponding extraction problem, and use the file and the extraction problem as the basic task carrier; Perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets; Establish a placeholder book for the file and the extraction problem, and verify whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid interval recorded in the placeholder book, and obtain the verification result. If the verification result indicates that there is overlap, the new candidate object corresponding to the new candidate interval will be rejected. If the verification result indicates no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, and cross-wheel and cross-segment deduplication and merging processing is performed based on the data of all the valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

[0005] In one possible implementation, the extraction of target information from the basic task carrier through multiple rounds and / or multiple slices to obtain extracted candidate data packets includes: Initialize the extraction rounds and fragment counters. The extraction rounds correspond to a preset upper limit for the extraction rounds, and the fragment counters correspond to a preset fragment division rule. Load the Transformer-based information extraction model and configuration parameters, including the extraction range and the context window size. The extraction operation is performed sequentially according to the round and / or slice. After the extraction of each round and / or slice is completed, candidate data containing extraction information, hit interval coordinates, and model confidence are obtained. Extract the context window corresponding to the hit interval coordinates from the candidate data, and associate the context window with the identifier of the corresponding round and / or slice to generate a complete candidate data packet.

[0006] In one possible implementation, the step of establishing a placeholder book for the file and the extraction problem, and verifying whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid intervals recorded in the placeholder book, to obtain the verification result, includes: Check whether a placeholder book for the intervals corresponding to the file and the extraction problem has been established; If the interval placeholder book is not established, the interval placeholder book is initialized with the extraction problem as the core dimension. The interval placeholder book contains key information fields such as the coordinates of the hit interval of the valid candidate, the entry timestamp, and the round / fragment identifier. If the interval placeholder book has been established, the coordinates of the hit interval are directly extracted from the candidate data packet. All existing valid intervals recorded in the interval placeholder book are traversed, and an interval overlap verification algorithm is used to verify whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates and the existing valid interval, and the verification result is obtained. The verification operation corresponding to the overlap verification algorithm includes calculating the intersection length ratio of the new candidate interval and the existing valid interval, and determining that it is an overlap when the intersection length ratio is greater than a preset threshold.

[0007] In one possible implementation, the step of rejecting the new candidate object corresponding to the new candidate interval if the verification result indicates overlap includes: If there is an overlap between the new candidate interval and the existing valid interval, the new candidate object corresponding to the new candidate interval is marked as an invalid candidate object according to the first-come-first-served rule, and the result of failing the verification is fed back to the target information multi-round extraction execution module that performs the extraction operation, so that the invalid candidate object is rejected and not included in subsequent processing. In one possible implementation, if the verification result indicates no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, including: If there is no overlap between the new candidate interval and the existing valid interval, the coordinates of the hit interval, the entry timestamp, and the round / fragment identifier are written to the interval placeholder book according to the first-come-first-served rule to complete the interval placeholder locking. The new candidate object corresponding to the new candidate interval is marked as a valid candidate object. The context window and model confidence of the valid candidate object are synchronized to the statistical cache pool, and the result of passing the verification is fed back to the target information multi-round extraction execution module. The target information multi-round extraction execution module records the valid candidate objects based on the results of the verification and skips the invalid candidate objects based on the results of the verification failure. It increments the extraction round and fragment counters and repeats the extraction and verification operations until the preset extraction round / fragment upper limit threshold is reached. Finally, it outputs and summarizes the data of all the valid candidate objects.

[0008] In one possible implementation, the cross-round and cross-segment deduplication and merging process based on the data of all the valid candidate objects to obtain the final cross-round and cross-segment deduplication and merging result includes: Based on the data of all the valid candidate objects, the data support information corresponding to the valid candidate objects is retrieved from the statistical cache pool. The data support information includes the target context window, target model confidence, frequency of occurrence of the same candidate object in files of different rounds / slices, and global frequency of occurrence of the same candidate object in the same extraction problem of the same type of file. Based on the data support information, all the valid candidate objects are sorted according to a specified priority rule to obtain a sorting result. The specified priority rule includes: using a descending order of the frequency of occurrence within the file and the global frequency of occurrence as the primary sorting dimension, so that the valid candidate objects with higher occurrence frequencies are given higher priority; when the occurrence frequencies are the same, using a descending order of the combined weight of the target model confidence and text consistency as the secondary sorting dimension to optimize the sorting result. Based on the sorting results, perform deduplication and merging processing on all the valid candidate objects: Based on the ranking result, the first valid candidate object is determined as the core claim object, and the other valid candidate objects are determined as supplementary trace data, generating a complete data set of core claims combined with supplementary trace data. The complete dataset is supplemented with frequency statistics of occurrence, sorting basis data of sorting results, and contextual trace data of each valid candidate object to make the results traceable and obtain integrated data. The integrated data is standardized according to a specified format to obtain a deduplicated single core claim and associated target trace data. The target trace data is used for business system calls and review, so as to trace the extraction background of each candidate object through the target trace data.

[0009] In one possible implementation, the step of verifying whether there is overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book further includes: In the process of verifying whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book, while judging the interval overlap, the weight of the model confidence of the new candidate object and the weight of the text consistency of the earlier candidate object are calculated, wherein the text consistency is the semantic similarity of the two candidate extracted information calculated by the cosine similarity algorithm. When the overall weight of a later-arriving candidate is higher than the specified difference of an earlier-arriving candidate, a rule adjustment is triggered, and the later-arriving candidate replaces the placeholder in the interval occupied by the earlier-arriving candidate. The relevant data in the statistical cache pool is also updated to enable flexible rule adaptation.

[0010] Secondly, this application provides a cross-wheel and cross-plate deduplication and merging device, comprising: The acquisition unit is used to acquire the same file to be processed and the corresponding extraction problem, and to use the file and the extraction problem as the basic task carrier; The extraction unit is used to perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets; The verification unit is used to establish a placeholder book for the intervals corresponding to the file and the extraction problem, and to verify whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid intervals recorded in the placeholder book, and to obtain the verification result. The rejection unit is used to reject the new candidate object corresponding to the new candidate interval if the verification result indicates that there is overlap. The deduplication and merging unit is used to write the coordinates of the hit interval to the interval placeholder book if the verification result is that there is no overlap, so as to complete the cross-wheel and cross-segment interval placeholder locking, determine the new candidate object as a valid candidate object, synchronize the context window of the valid candidate object, and perform cross-wheel and cross-segment deduplication and merging processing based on the data of all the valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

[0011] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0012] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0013] This application brings the following beneficial effects: This application provides a method, apparatus, and electronic device for cross-processing and cross-segment deduplication and merging. It can acquire the same file to be processed and the corresponding extraction problem, and use the file and extraction problem as a basic task carrier. It performs multi-round and / or multi-segment extraction operations on the basic task carrier to obtain extracted candidate data packets. It establishes a placeholder book for the intervals corresponding to the file and the extraction problem, and verifies whether there is overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packets and the existing valid intervals recorded in the placeholder book, obtaining a verification result. If the verification result indicates overlap, the new candidate object corresponding to the new candidate interval is rejected. If the verification result indicates no overlap, the hit interval coordinates are written to the placeholder book to complete cross-processing and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, and cross-processing and cross-segment deduplication and merging processing is performed based on the data of all valid candidate objects to obtain the final cross-processing and cross-segment deduplication and merging result. In this solution, for the same... A placeholder interval book is established for files and issues of the same type, creating an interval placeholder structure at the issue level. This enables global interval control over extraction results for the same file and issue, forming a constraint on the correlation of extraction results in text position. When a newly hit interval overlaps with an already occupied interval, the subsequent hit is rejected according to the first-come-first-served rule. This implements a first-come-first-served cross-round and cross-segment extraction hit deduplication and merging method, thereby eliminating duplicate hits at the source and preventing their generation. At the same time, the adjudication of conflicting results achieves a systematic solution that balances recall and stability, forming a system for handling duplicates with existing intervals. The algorithm rejects candidate data based on a first-come-first-served rule and outputs a final candidate set after contextual tracking and frequency statistics. In multi-round extraction scenarios, it resolves duplicate and conflicting results, ultimately obtaining a single and stable claim. This avoids redundancy and difficulty in resolving duplicates and conflicts in multi-round / multi-slice extraction results, thereby improving information utilization efficiency. It can significantly reduce manual review costs while ensuring information recall, expand the application of information extraction technology in high-reliability scenarios, improve the reliability and practical value of information extraction results, and solve the technical problem of low reliability of information extraction results.

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the cross-wheel and cross-piece deduplication and merging method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the execution of each module in the cross-wheel and cross-slice deduplication and merging method provided in the embodiments of this application; Figure 3 This is another flowchart illustrating the cross-wheel and cross-piece deduplication and merging method provided in the embodiments of this application; Figure 4 A flowchart illustrating the execution of the target information multi-round extraction and merging module in the cross-round and cross-slice deduplication and merging method provided in this application embodiment; Figure 5 A flowchart illustrating the execution of the cross-wheel and cross-piece interval occupancy locking module in the cross-wheel and cross-piece deduplication and merging method provided in this application embodiment; Figure 6 A flowchart illustrating the execution of the deduplication and merging result output module in the cross-wheel and cross-piece deduplication and merging method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a cross-wheel and cross-plate deduplication and merging device provided in an embodiment of this application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] Currently, existing technologies mainly employ three types of solutions for deduplication and conflict resolution in multi-round / multi-slice extraction: First, post-processing filtering based on text similarity, which eliminates duplicates by calculating the string matching degree or semantic similarity of the extraction results; second, rule-based conflict resolution, which filters conflict results by pre-setting domain-specific rules (such as priority ranking and keyword weight); and third, optimization based on model confidence, which selects the candidate result with the highest confidence score as the final output based on the confidence score output by the extraction model. However, these existing technologies have significant drawbacks: First, similarity filtering methods struggle to handle semantically equivalent but significantly different repetitive information, and are prone to missed detections or misjudgments due to improper threshold settings, making them unsuitable for complex overlapping scenarios across rounds and segments; Second, rule-based adjudication methods rely on manually preset rules, resulting in poor generalization ability and requiring repeated adaptation when dealing with different domains and types of text, leading to high maintenance costs; Third, confidence-based selection methods only focus on the reliability of single-round extraction results, ignoring the positional relevance and global frequency of information in the text, making them susceptible to single-round prediction errors and resulting in insufficient stability of the conclusions.

[0020] Existing methods struggle to prevent duplicate hits at the source, and the lack of systematic constraints in conflict resolution leads to redundant duplicate entries and difficulty in resolving conflicts, resulting in insufficient stability of conclusions and low information utilization efficiency. Furthermore, existing technologies have weak generalization capabilities, high maintenance costs, and are prone to missed detections and misjudgments due to threshold settings and model errors, significantly increasing the time and manpower costs of manual review.

[0021] Based on this, embodiments of this application provide a method, apparatus, and electronic device for deduplication and merging across wheels and slices, which can solve technical problems such as low reliability of information extraction results.

[0022] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a method for deduplication and merging across wheels and slices provided in an embodiment of this application. Figure 1 As shown, the method includes: Step S110: Obtain the same file to be processed and the corresponding extraction problem, and use the file and extraction problem as the basic task carrier.

[0024] For example, the system receives the same file to be processed and its corresponding target extraction question, inputting it as a whole task into the cross-processing and cross-segment interval placeholder locking module and the target information multi-round extraction execution module. This provides the basic task carrier for subsequent cross-processing deduplication and merging. The file to be processed can cover various text carriers requiring key information extraction, such as contracts, tenders, and reports. The target extraction question must be clearly defined, such as "extracting the contract signing date from the document" or "identifying the project budget amount," to ensure the accuracy of the extraction task. For example, ... Figure 2 As shown, the system corresponding to the cross-wheel and cross-segment deduplication and merging method includes: a cross-wheel and cross-segment interval occupancy locking module; a target information multi-round extraction execution module; and a deduplication and merging result output module.

[0025] Step S120: Perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain the extracted candidate data packets.

[0026] As one possible implementation, the process of performing multiple rounds and / or multiple fragments of target information extraction operations on the basic task carrier to obtain extracted candidate data packets may specifically include the following steps: Initialize the extraction rounds and fragment counters. The extraction rounds correspond to a preset upper limit for extraction rounds, and the fragment counters correspond to preset fragment division rules. Load the Transformer-based information extraction model and configuration parameters, including the extraction range and context window size. Extraction operations are performed sequentially according to rounds and / or slices. After extraction of each round and / or slice, candidate data containing extraction information, hit interval coordinates, and model confidence are obtained. The context window corresponding to the hit interval coordinates is extracted from the candidate data, and the context window is associated with the corresponding round and / or slice identifier to generate a complete candidate data package.

[0027] For example, such as Figure 3 and Figure 4As shown, the multi-round target information extraction execution module is started to perform multi-round / multi-segment target information extraction operations on the same file and the same target extraction problem. First, the extraction round and segment counters are initialized, and a preset extraction round limit (e.g., 5 rounds) or segment division rule (e.g., segmentation by document chapter) is set. At the same time, the adapted extraction model (e.g., information extraction model based on Transformer) and configuration parameters are loaded, including the extraction range (full text or specified chapter), context window size (e.g., 20 characters before and after the hit interval), etc. Then, extraction is performed sequentially by round / segment. After each round / segment extraction is completed, candidate data containing extraction information, hit interval coordinates (e.g., document page number-paragraph number-character start / end position), and model confidence is obtained. The context window of the hit interval is extracted, associated with the corresponding round / segment identifier, and a complete candidate data package is formed and output to the cross-round and cross-segment interval placeholder locking module.

[0028] Step S130: Establish a placeholder book for the intervals corresponding to the file and the extraction problem, and verify whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid intervals recorded in the placeholder book, and obtain the verification result.

[0029] As an example, the above-mentioned process of establishing a placeholder book for the intervals corresponding to the extraction problem, and verifying whether there is any overlap between the new candidate intervals corresponding to the coordinates of the hit intervals in the candidate data packets and the existing valid intervals already recorded in the placeholder book, and obtaining the verification results, may specifically include the following steps: Check if a placeholder book for the corresponding file and extraction question has been created; if no placeholder book has been created, initialize the placeholder book with the extraction question as the core dimension. The placeholder book contains key information fields such as the coordinates of the hit interval of the valid candidate, the entry timestamp, and the round / fragment identifier. If a placeholder book for the interval has been established, the coordinates of the hit interval are directly extracted from the candidate data packet. All existing valid intervals recorded in the placeholder book are traversed, and the interval overlap verification algorithm is used to verify whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates and the existing valid intervals, and the verification result is obtained. The verification operation corresponding to the overlap verification algorithm includes calculating the intersection length ratio of the new candidate interval and the existing valid intervals, and determining that it is an overlap when the intersection length ratio is greater than a preset threshold.

[0030] For example, such as Figure 3 and Figure 5As shown, after receiving the candidate data packet output by the target information multi-round extraction execution module, the cross-round and cross-segment interval placeholder locking module first checks whether an interval placeholder book corresponding to the target extraction problem has been established. If not, it immediately initializes the interval placeholder book with the target extraction problem as the core dimension. This book must contain key information fields such as the coordinates of the hit interval of the valid candidate, the entry timestamp, and the round / segment identifier. If it has been established, it directly extracts the coordinates of the hit interval from the candidate data packet. Then, it traverses all the valid intervals recorded in the interval placeholder book and uses an interval overlap verification algorithm (such as calculating the intersection length ratio of two intervals, and determining it as an overlap when the ratio is greater than a preset threshold of 50%) to verify whether the new candidate interval partially or completely overlaps with the existing valid intervals.

[0031] Optionally, the above method directly extracts the coordinates of the hit interval from the candidate data packet, traverses all existing valid intervals recorded in the interval placeholder book, and uses an interval overlap verification algorithm to perform a verification operation on whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates and the existing valid intervals, and obtains the verification result. Specifically, it may include the following steps: The coordinates of the hit interval are directly extracted from the candidate data packets. All existing valid intervals recorded in the interval placeholder book are traversed. An interval overlap verification algorithm is used to calculate the percentage of overlap between the new candidate interval corresponding to the hit interval coordinates and the existing valid intervals using the following formula. A verification operation is then performed based on this percentage of overlap to obtain the verification result.

[0032] in, The intersection length ratio represents the overlap rate between the new candidate interval and the existing valid interval; This indicates the end position of the new candidate interval; This indicates the end position of the existing valid interval; Indicates the starting position of the new candidate interval; This indicates the starting position of the existing valid interval.

[0033] In this embodiment of the application, the intersection length ratio between the new candidate interval corresponding to the hit interval coordinate and the existing valid interval is calculated by the above calculation formula, which can more accurately determine the overlap rate and improve the accuracy of the verification results.

[0034] In an optional implementation, the verification of whether there is overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book may specifically include the following steps: In the process of verifying whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book, while judging the interval overlap, the weights of the model confidence of the new candidate object and the first candidate object and the weights of text consistency are calculated. Among them, text consistency is the semantic similarity of the extracted information of two candidates calculated by the cosine similarity algorithm. When the overall weight of a later candidate is higher than the specified difference of an earlier candidate, the rule is adjusted and the later candidate replaces the placeholder in the interval occupied by the earlier candidate. The relevant data in the statistical cache pool is also updated to make the rule flexible and adaptable.

[0035] In practical applications, such as Figure 3 As shown, for scenarios requiring enhanced flexibility, a preferred solution can be enabled, introducing weighted optimization based on the first-to-arrive locking rule. When performing overlap verification, the cross-wheel / cross-segment interval occupancy locking module not only determines interval overlap but also calculates the model confidence weight and text consistency weight between the new candidate and the first-arrive candidate. Text consistency is calculated using a cosine similarity algorithm to determine the semantic similarity of the extracted information from the two candidates. When the combined weight of the later-arrive candidate (e.g., model confidence accounting for 60% and text consistency accounting for 40%) is significantly higher than that of the first-arrive candidate (e.g., the difference exceeds a preset threshold of 0.3), rule adjustment can be triggered, allowing the later-arrive candidate to replace the first-arrive candidate in the interval occupancy book. Simultaneously, relevant data in the statistical cache pool is updated, achieving flexible rule adaptation. Weighting the "first-to-arrive" strategy based on model confidence and text consistency is supported.

[0036] Step S140: If the verification result indicates that there is overlap, the new candidate object corresponding to the new candidate interval is rejected.

[0037] In an optional implementation, the above-mentioned rejection of new candidate objects corresponding to new candidate intervals if the verification result shows overlap can specifically include the following steps: if there is overlap between the new candidate interval and the existing valid interval, the new candidate objects corresponding to the new candidate interval are marked as invalid candidate objects according to the first-come-first-served rule, and the result of failing the verification is fed back to the target information multi-round extraction execution module that performs the extraction operation, so that the invalid candidate objects are rejected and not included in subsequent processing.

[0038] For example, such as Figure 3 As shown, if the verification finds that the new candidate interval overlaps with the existing interval, the new candidate is marked as invalid data according to the first-come-first-served rule. The result of "failed verification" is fed back to the target information multi-round extraction execution module. The invalid candidate will be rejected and will not be included in subsequent processing.

[0039] Step S150: If the verification result is that there is no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, and the cross-wheel and cross-segment deduplication and merging process is performed based on the data of all valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

[0040] As an example, if the verification result shows no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, which may include the following steps: If there is no overlap between the new candidate interval and the existing valid interval, the coordinates of the hit interval, the entry timestamp and the round / fragment identifier are written to the interval placeholder book according to the first-come-first-served rule to complete the interval placeholder locking. The new candidate object corresponding to the new candidate interval is marked as a valid candidate object. The context window and model confidence of the valid candidate object are synchronized to the statistical cache pool, and the result of passing the verification is fed back to the target information multi-round extraction execution module. The target information multi-round extraction execution module records valid candidate objects based on the results of successful verification and skips invalid candidate objects based on the results of unsuccessful verification. It increments the extraction round and fragment counters and repeats the extraction and verification process until the preset extraction round / fragment upper limit threshold is reached. Finally, it outputs and summarizes the data of all valid candidate objects.

[0041] In practical applications, such as Figure 3 and Figure 6 As shown, if a new candidate interval does not overlap with an existing interval, its matched interval coordinates, entry timestamp, and round / fragment identifier are written to the interval placeholder book to complete interval placement locking. Simultaneously, the candidate is marked as valid data, and its context window, model confidence, and other information are synchronized to the statistical cache pool. The "passed verification" result is then fed back to the target information multi-round extraction execution module. After receiving the feedback, the target information multi-round extraction execution module records valid candidates and skips invalid ones. It then increments the round / fragment counter and repeats the extraction and verification process until the preset extraction round / fragment limit is reached. Finally, all valid candidate data is aggregated and output to the deduplication and merging result output module.

[0042] As an optional implementation, the above-mentioned deduplication and merging process based on the data of all valid candidate objects, across rounds and across segments, to obtain the final deduplication and merging result, may specifically include the following steps: Based on the data of all valid candidate objects, retrieve the data support information corresponding to the valid candidate objects from the statistical cache pool. The data support information includes the target context window, target model confidence, frequency of occurrence of the same candidate object in files of different rounds / slices, and global frequency of occurrence of the same candidate object in the same extraction problem of the same type of file. Based on the data support information, all valid candidate objects are sorted according to the specified priority rules to obtain the sorting results. The specified priority rules include: using the descending order of occurrence frequency in the file and global occurrence frequency as the primary sorting dimension, so that the valid candidate objects with higher occurrence frequency are given higher priority; when the occurrence frequency is the same, the descending order of the comprehensive weight of the target model confidence and text consistency is used as the secondary sorting dimension to optimize the sorting results. Based on the ranking results, a deduplication and merging process is performed on all valid candidate objects: the valid candidate object that retains the first place in the ranking is determined as the core claim object, and the valid candidate objects other than the core claim object are determined as supplementary trace data, generating a complete data set of core claims combined with supplementary trace data; To supplement the complete dataset with frequency statistics of occurrences, sorting basis data for the sorting results, and contextual trace data for each valid candidate object, so as to make the results traceable and obtain the integrated and processed data; The integrated data is standardized according to a specified format to obtain a deduplicated single core claim and associated target trace data. The target trace data is used for business system calls and review, so as to trace the extraction background of each candidate object through the target trace data.

[0043] For example, such as Figure 3 As shown, the deduplication and merging result output module receives the target information. After the multi-round extraction execution module summarizes all valid candidate data, it retrieves the corresponding data support information from the statistical cache pool, including the context window of each valid candidate, the frequency of occurrence within the file (the number of times the same candidate appears in different rounds / slices), and the global candidate frequency (the number of times the candidate appears in the same type of file and the same problem extraction). Then, the candidate is sorted according to preset rules: the primary sorting dimension is the descending order of the frequency of candidates within the file plus the global candidate frequency, ensuring that candidates with higher frequencies are given priority; when the frequencies are the same, the secondary sorting dimension is the descending order of the comprehensive weight of model confidence and text consistency, further optimizing the sorting results.

[0044] Then, based on the ranking results, deduplication and merging are performed, retaining the top-ranked candidate as the core claim, and the remaining valid candidates as supplementary trace data, forming a complete dataset of "core claim + supplementary trace data". This dataset is then integrated, supplementing it with frequency statistics reports, ranking criteria explanations, and contextual trace details for each candidate to ensure traceability. Finally, the integrated data is standardized according to a preset format (such as JSON, XML, or a custom structured format), outputting a deduplicated and stable single core claim and associated trace data for use by downstream business systems or manual review. During manual review, the trace data allows for quick tracing of the extraction context of each candidate, significantly reducing review costs.

[0045] By recording and retaining the context window of the hit and counting the frequency of candidates within the file and globally, a ranking is formed. While ensuring recall, the system outputs deduplicated and stable conclusions, thereby reducing the cost of manual review.

[0046] In this embodiment, a placeholder interval book is established for the same file and the same issue, creating an interval placeholder structure at the issue level. This enables global interval control over the extraction results for the same file and the same issue, forming a constraint on the correlation of the extraction results in the text position. When a newly hit interval overlaps with an already occupied interval, the subsequent hit is rejected according to the first-come-first-served rule. This achieves a first-come-first-served cross-round and cross-segment extraction hit deduplication and merging method, thereby eliminating duplicate hits from the source and avoiding their generation. At the same time, the adjudication of conflicting results balances recall and stability. The systematic approach rejects candidate claims that overlap with existing intervals based on a first-come-first-served rule and outputs a final candidate set that has been processed by contextual tracking and frequency statistics. In multi-round extraction scenarios, it resolves duplicate and conflicting results, ultimately obtaining a single and stable claim. This avoids redundancy and difficulty in resolving duplicates and conflicts in multi-round / multi-segment extraction results, thereby improving information utilization efficiency. While ensuring information recall, it significantly reduces the cost of manual review, expands the application of information extraction technology in scenarios with high reliability requirements, and enhances the reliability and practical value of information extraction results.

[0047] Figure 7 A schematic diagram of a cross-wheel, cross-plate deduplication and merging device is provided. (See diagram below.) Figure 7 As shown, the cross-wheel and cross-plate de-duplication and merging device 700 includes: The acquisition unit 701 is used to acquire the same file to be processed and the corresponding extraction problem, and to use the file and the extraction problem as the basic task carrier. Extraction unit 702 is used to perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets; The verification unit 703 is used to establish an interval placeholder book corresponding to the file and the extraction problem, and to verify whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid interval recorded in the interval placeholder book, and to obtain the verification result. The rejection unit 704 is used to reject the new candidate object corresponding to the new candidate interval if the verification result indicates that there is overlap. The deduplication and merging unit 705 is used to write the coordinates of the hit interval to the interval placeholder book if the verification result is that there is no overlap, so as to complete the cross-wheel and cross-segment interval placeholder locking, determine the new candidate object as a valid candidate object, synchronize the context window of the valid candidate object, and perform cross-wheel and cross-segment deduplication and merging processing based on the data of all the valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

[0048] The cross-wheel and cross-piece deduplication and merging device provided in this application has the same technical features as the cross-wheel and cross-piece deduplication and merging method provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0049] An electronic device provided in this application embodiment, such as Figure 8 As shown, the electronic device 800 includes a processor 802 and a memory 801. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0050] See Figure 8 The electronic device also includes a bus 803 and a communication interface 804. The processor 802, the communication interface 804 and the memory 801 are connected through the bus 803. The processor 802 is used to execute executable modules, such as computer programs, stored in the memory 801.

[0051] The memory 801 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 804 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0052] Bus 803 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0053] The memory 801 is used to store programs. After receiving an execution instruction, the processor 802 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 802 or implemented by the processor 802.

[0054] The processor 802 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 802 or by instructions in software form. The processor 802 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 801, and processor 802 reads the information from memory 801 and, in conjunction with its hardware, completes the steps of the above method.

[0055] Corresponding to the above-described cross-wheel and cross-segment deduplication and merging method, this application embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and run by a processor, the computer-executable instructions cause the processor to perform the steps of the above-described cross-wheel and cross-segment deduplication and merging method.

[0056] The cross-wheel and cross-piece deduplication and merging device provided in this application embodiment can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0057] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0058] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0060] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0061] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the cross-process deduplication and merging method described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0063] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for deduplication and merging across wheels and pieces, characterized in that, The method includes: Obtain the same file to be processed and the corresponding extraction problem, and use the file and the extraction problem as the basic task carrier; Perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets; Establish a placeholder book for the file and the extraction problem, and verify whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid interval recorded in the placeholder book, and obtain the verification result. If the verification result indicates that there is overlap, the new candidate object corresponding to the new candidate interval will be rejected. If the verification result indicates no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-segment interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, and cross-wheel and cross-segment deduplication and merging processing is performed based on the data of all the valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

2. The method according to claim 1, characterized in that, The step of performing multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets includes: Initialize the extraction rounds and fragment counters. The extraction rounds correspond to a preset upper limit for the extraction rounds, and the fragment counters correspond to a preset fragment division rule. Load the Transformer-based information extraction model and configuration parameters, including the extraction range and the context window size. The extraction operation is performed sequentially according to the round and / or slice. After the extraction of each round and / or slice is completed, candidate data containing extraction information, hit interval coordinates, and model confidence are obtained. Extract the context window corresponding to the hit interval coordinates from the candidate data, and associate the context window with the identifier of the corresponding round and / or slice to generate a complete candidate data packet.

3. The method according to claim 2, characterized in that, The process involves establishing a placeholder book for the file and the extraction problem, and verifying whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid intervals recorded in the placeholder book, to obtain the verification result, including: Check whether a placeholder book for the intervals corresponding to the file and the extraction problem has been established; If the interval placeholder book is not established, the interval placeholder book is initialized with the extraction problem as the core dimension. The interval placeholder book contains key information fields such as the coordinates of the hit interval of the valid candidate, the entry timestamp, and the round / fragment identifier. If the interval placeholder book has been established, the coordinates of the hit interval are directly extracted from the candidate data packet. All existing valid intervals recorded in the interval placeholder book are traversed, and an interval overlap verification algorithm is used to verify whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates and the existing valid interval, and the verification result is obtained. The verification operation corresponding to the overlap verification algorithm includes calculating the intersection length ratio of the new candidate interval and the existing valid interval, and determining that it is an overlap when the intersection length ratio is greater than a preset threshold.

4. The method according to claim 3, characterized in that, The step of rejecting new candidate objects corresponding to the new candidate interval if the verification result indicates overlap includes: If there is an overlap between the new candidate interval and the existing valid interval, the new candidate object corresponding to the new candidate interval is marked as an invalid candidate object according to the first-come-first-served rule, and the result of failing the verification is fed back to the target information multi-round extraction execution module that performs the extraction operation, so that the invalid candidate object is rejected and not included in subsequent processing.

5. The method according to claim 4, characterized in that, If the verification result indicates no overlap, the coordinates of the hit interval are written to the interval placeholder book to complete the cross-wheel and cross-piece interval placeholder locking, and the new candidate object is determined as a valid candidate object. The context window of the valid candidate object is synchronized, including: If there is no overlap between the new candidate interval and the existing valid interval, the coordinates of the hit interval, the entry timestamp, and the round / fragment identifier are written to the interval placeholder book according to the first-come-first-served rule to complete the interval placeholder locking. The new candidate object corresponding to the new candidate interval is marked as a valid candidate object. The context window and model confidence of the valid candidate object are synchronized to the statistical cache pool, and the result of passing the verification is fed back to the target information multi-round extraction execution module. The target information multi-round extraction execution module records the valid candidate objects based on the results of the verification and skips the invalid candidate objects based on the results of the verification failure. It increments the extraction round and fragment counters and repeats the extraction and verification operations until the preset extraction round / fragment upper limit threshold is reached. Finally, it outputs and summarizes the data of all the valid candidate objects.

6. The method according to claim 5, characterized in that, The deduplication and merging process based on the data of all the valid candidate objects across rounds and segments is performed to obtain the final deduplication and merging result across rounds and segments, including: Based on the data of all the valid candidate objects, the data support information corresponding to the valid candidate objects is retrieved from the statistical cache pool. The data support information includes the target context window, target model confidence, frequency of occurrence of the same candidate object in files of different rounds / slices, and global frequency of occurrence of the same candidate object in the same extraction problem of the same type of file. Based on the data support information, all the valid candidate objects are sorted according to a specified priority rule to obtain a sorting result. The specified priority rule includes: using a descending order of the frequency of occurrence within the file and the global frequency of occurrence as the primary sorting dimension, so that the valid candidate objects with higher occurrence frequencies are given higher priority; when the occurrence frequencies are the same, using a descending order of the combined weight of the target model confidence and text consistency as the secondary sorting dimension to optimize the sorting result. Based on the sorting results, perform deduplication and merging processing on all the valid candidate objects: Based on the ranking result, the first valid candidate object is determined as the core claim object, and the other valid candidate objects are determined as supplementary trace data, generating a complete data set of core claims combined with supplementary trace data. The complete dataset is supplemented with frequency statistics of occurrence, sorting basis data of sorting results, and contextual trace data of each valid candidate object to make the results traceable and obtain integrated data. The integrated data is standardized according to a specified format to obtain a deduplicated single core claim and associated target trace data. The target trace data is used for business system calls and review, so as to trace the extraction background of each candidate object through the target trace data.

7. The method according to claim 6, characterized in that, The step of verifying whether there is any overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book also includes: In the process of verifying whether there is an overlap between the new candidate interval corresponding to the hit interval coordinates in the candidate data packet and the existing valid intervals recorded in the interval placeholder book, while judging the interval overlap, the weight of the model confidence of the new candidate object and the weight of the text consistency of the earlier candidate object are calculated, wherein the text consistency is the semantic similarity of the two candidate extracted information calculated by the cosine similarity algorithm. When the overall weight of a later-arriving candidate is higher than the specified difference of an earlier-arriving candidate, a rule adjustment is triggered, and the later-arriving candidate replaces the placeholder in the interval occupied by the earlier-arriving candidate. The relevant data in the statistical cache pool is also updated to enable flexible rule adaptation.

8. A cross-wheel and cross-plate deduplication and merging device, characterized in that, include: The acquisition unit is used to acquire the same file to be processed and the corresponding extraction problem, and to use the file and the extraction problem as the basic task carrier; The extraction unit is used to perform multiple rounds and / or multiple slices of target information extraction operations on the basic task carrier to obtain extracted candidate data packets; The verification unit is used to establish a placeholder book for the intervals corresponding to the file and the extraction problem, and to verify whether there is any overlap between the new candidate interval corresponding to the coordinates of the hit interval in the candidate data packet and the existing valid intervals recorded in the placeholder book, and to obtain the verification result. The rejection unit is used to reject the new candidate object corresponding to the new candidate interval if the verification result indicates that there is overlap. The deduplication and merging unit is used to write the coordinates of the hit interval to the interval placeholder book if the verification result is that there is no overlap, so as to complete the cross-wheel and cross-segment interval placeholder locking, determine the new candidate object as a valid candidate object, synchronize the context window of the valid candidate object, and perform cross-wheel and cross-segment deduplication and merging processing based on the data of all the valid candidate objects to obtain the final cross-wheel and cross-segment deduplication and merging result.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images