Archive data structured extraction method based on machine learning

Abstract

The invention discloses an archive data structured extraction method based on machine learning, and the method comprises the following steps: carrying out the normalization processing, noise suppression and sequence segmentation of an archive text, and obtaining a processable text sequence; the method comprises the following steps: generating a plurality of candidate field fragments in a text sequence, extracting multi-level features for each fragment, carrying out joint evaluation on boundary combination and feature representation of the field fragments by introducing a minimum description length criterion, determining an optimal fragment division result, generating compressed feature representation, and constructing sequence input by taking the field fragments as an integral unit. And utilizing a semi-Markov conditional random field model to carry out overall modeling and joint decoding on the variable-length field to obtain a corresponding relationship between the field tag and the field fragment, and finally generating a structured archive data result. According to the method, the dependence on manual rules can be reduced, and the accuracy and adaptability of structured extraction of the archive data are improved.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and in particular to a method for structured extraction of archival data based on machine learning. Background Technology

[0002] With the advancement of archival digitization and intelligent archival management, massive amounts of historical archives, business archives, and electronic documents are gradually being centrally stored and utilized in the form of scanned text or electronic text. Automatic organization, cataloging, and structured extraction of archival content have become important application scenarios in archival informatization. Currently, archival data is typically in the form of unstructured text, requiring the identification, segmentation, and classification of key information within the text content to form searchable and analyzable structured archival data.

[0003] Existing technologies for structuring archival data still have significant shortcomings. On the one hand, traditional methods often rely on manual rules, template matching, or fixed field patterns for processing. Rule design and maintenance are costly, and when archival types, formats, or historical periods change, errors in field identification, boundary misalignment, or omissions in extraction can easily occur, making them ill-suited to complex and diverse archival texts. On the other hand, automatic extraction methods based on conventional sequence labeling models typically use single characters or fixed windows as modeling units, making it difficult to directly process variable-length fields. Field boundary identification and feature construction processes are disconnected, and errors easily accumulate in multiple processing stages, leading to insufficient integrity and consistency in the structured results. Archival texts commonly contain noise such as optical recognition errors, non-standard layout, and mixed symbols. Existing methods lack effective mechanisms for distinguishing and suppressing noise features, making them susceptible to interference from irrelevant information and further reducing the accuracy of structured extraction.

[0004] Therefore, how to provide a method for structured extraction of archival data based on machine learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a machine learning-based method for structured extraction of archival data. This invention achieves automatic conversion of unstructured archival content into structured archival data by performing fragment-level modeling and sequence analysis on archival text. This invention comprehensively utilizes text feature modeling, information theory criteria, and sequence modeling techniques to uniformly process field fragments, field boundaries, and feature representations in archival text, completing field identification and field value generation. It can achieve structured extraction of complex archival text with reduced reliance on manual rules, possessing advantages such as strong adaptability, high structural consistency, and high processing accuracy.

[0006] The method for structured extraction of archival data based on machine learning according to embodiments of the present invention includes the following steps: Obtain the archival text data, and perform character normalization, noise character suppression, and serialization segmentation on the archival text data to obtain the text sequence to be processed; A candidate field fragment set is generated based on the text sequence to be processed. The candidate field fragment set consists of multiple candidate field fragments covering a preset length range, and each candidate field fragment has a fragment start and end boundary. For each candidate field fragment in the candidate field fragment set, a multi-granularity feature set is extracted, which includes character-level features, position-level features, and contextual relationship features; Using the minimum description length criterion as the unified objective function, the fragment boundary combination formed by the start and end boundaries of each candidate field fragment is jointly encoded and evaluated with the corresponding multi-granularity feature set, outputting the fragment partitioning result with the minimum description length, and generating a compressed feature representation for each field fragment in the fragment partitioning result; Based on the field fragments and compressed feature representations in the fragment partitioning results, a fragment sequence input for a semi-Markov conditional random field model is constructed, and a set of field labels is defined to form a fragment-level sequence labeling task. The model parameters are obtained by training a semi-Markov conditional random field model, and fragment-level joint decoding is performed on the fragment sequence input to output the correspondence between the field label set and the field fragment; Based on the correspondence between the field label set and the field fragment, the field fragment is mapped to structured field values ​​to generate structured archive data results.

[0007] Optionally, the generation of the text sequence to be processed includes: acquiring archival text data; performing character normalization processing on the archival text data character by character to form a normalized character sequence; the character normalization processing is used to uniformly map full-width characters and half-width characters, uppercase and lowercase characters, traditional characters and simplified characters, homographs with different codes, and preset variant characters to normalized characters; performing noise character suppression processing on the normalized character sequence to form a denoised character sequence; the noise character suppression processing is used to identify and delete non-semantic characters generated by optical character recognition errors, scanning interference, or encoding anomalies, and to merge consecutive blank characters and invalid symbols; performing serialization segmentation processing on the denoised character sequence; determining the initial segmentation position set based on line breaks, page breaks, tabs, and punctuation separators; adjusting the initial segmentation position set in combination with the continuity constraints of number strings, date strings, and number strings; and organizing the denoised character sequence into the text sequence to be processed based on the adjusted segmentation position set in the original order.

[0008] Optionally, the generation of the candidate field fragment set includes: Based on the text sequence to be processed, character position identifiers are established. Each character in the text sequence to be processed is assigned a continuous position identifier according to the order of its appearance in the text sequence to be processed, and a character sequence with position identifiers is obtained. Based on the character sequence with position identifier, a preset length range for candidate field segments is set. The preset length range is limited by a minimum length value and a maximum length value. The minimum length value limits the candidate field segment to contain one character, and the maximum length value limits the number of characters contained in the candidate field segment to not exceed a preset upper limit. Based on a preset length range, any position identifier in the character sequence with position identifier is used as the starting position of the segment. Under the condition that the preset length range is met and the end of the character sequence with position identifier is not exceeded, characters are continuously selected from the starting position of the segment to determine the ending position of the segment, generating a candidate field segment consisting of the character corresponding to the starting position of the segment to the character corresponding to the ending position of the segment. After generating candidate field fragments, record the start and end boundaries of the fragments. The start and end boundaries of the fragments are composed of the start position and the end position of the fragments, and establish a correspondence between the start and end boundaries of the fragments and the content of the candidate field fragments. Repeatedly perform the following steps for determining the start position of a segment, determining the end position of a segment, generating candidate field segments, and recording the start and end boundaries of segments for all position identifiers in a character sequence with position identifiers. This process is then used to obtain all candidate field segments and their start and end boundaries, forming a set of candidate field segments.

[0009] Optionally, the extraction of the multi-granularity feature set includes: for each candidate field fragment in the candidate field fragment set, reading the start and end boundaries of the fragment and determining the character sequence inside the fragment and its left and right adjacent context windows; extracting character-level features based on the character sequence inside the fragment, including character category features, character normalized morphology features, prefix and suffix substring features of the character sequence, and punctuation distribution features inside the fragment; extracting position-level features based on the position of the fragment start and end boundaries in the character sequence with position indicators, including the relative inline position of the fragment start position and the fragment end position, whether it crosses lines, fragment length range indicators, and distance indicators between the fragment and the nearest separator; extracting context relationship features based on the left and right adjacent context windows, including the character sequence features adjacent to the left side of the fragment, the character sequence features adjacent to the right side of the fragment, the combination features of the fragment and the adjacent separator type, and the boundary relative relationship features between the fragment and adjacent candidate field fragments; and indexing and encoding the character-level features, position-level features, and context relationship features according to a preset feature template to obtain the multi-granularity feature set of the candidate field fragment.

[0010] Optionally, the generation of compressed feature representations includes: Based on the candidate field fragment set, a fragment boundary combination set is constructed. The fragment boundary combination set consists of multiple fragment boundary combinations. Each fragment boundary combination is composed of the start and end boundaries of the candidate field fragments and corresponds to a set of field fragments that are arranged in order and do not overlap in the text sequence to be processed. For each fragment boundary combination in the fragment boundary combination set, read the corresponding candidate field fragments in the candidate field fragment set for each field fragment in the fragment boundary combination, and perform indexed encoding on the multi-granularity feature set of each field fragment according to the preset feature template to form a fragment-level feature sequence corresponding to the fragment boundary combination. Based on the fragment-level feature sequence, the start and end boundaries of each field fragment in the fragment boundary combination are used to determine the range of character positions covered by the field fragment. Then, noise mask inference is performed on the characters within the range of the field fragment and their feature dimensions in the fragment-level feature sequence to obtain the noise mask. Based on the noise mask, the fragment-level feature sequence is divided into content feature sequence and noise feature sequence. Based on content feature sequences and noise feature sequences, a joint coding structure is constructed and mapped. The joint coding structure includes a content coding structure and a noise coding structure. The content coding structure is used to establish a mapping between content feature values ​​and content coding symbols, and the noise coding structure is used to establish a mapping between noise feature values ​​and noise coding symbols. The mapping encodes the content feature sequence into a content coding sequence and the noise feature sequence into a noise coding sequence. Based on the content coding sequence and the noise coding sequence, the total description length corresponding to the segment boundary combination and the local description length of each field segment are calculated. The total description length is obtained by summing the local description lengths of each field segment in the segment boundary combination. The local description length of each field segment is composed of the description length of the content coding sequence and the description length of the noise coding sequence corresponding to the field segment. A dynamic programming solution structure is constructed based on the local description length. The character position identifiers in the text sequence to be processed are used as dynamic programming nodes, candidate field fragments are used as transitions between nodes, and the local description length of the corresponding field fragment is used as the transition cost. Through dynamic programming, the transition path with the minimum total description length is solved among all legal transition paths corresponding to the fragment boundary combination set, and the fragment boundary combination with the minimum total description length is generated as the fragment partitioning result. Output the segmentation result and the content encoding sequence and noise encoding sequence corresponding to each field segment in the segmentation result, and form the compressed feature representation of each field segment in the segmentation result based on the content encoding sequence and noise encoding sequence.

[0011] Optionally, the construction of the fragment sequence input includes: Based on the segmentation results, the start and end boundaries of each field segment in the segmentation results are read, and the field segments are sorted according to the order of the start position of the field segments in the text sequence to be processed, forming an ordered sequence of field segments; For each field segment in the ordered sequence of field segments, read the compressed feature representation corresponding to the field segment, and bind the compressed feature representation to the start and end boundaries of the field segment to form a segment-level observation item with the field segment as the basic unit; Based on fragment-level observations, the fragment-level observations are arranged sequentially according to the ordered sequence of field fragments to construct a fragment sequence structure; The fragment sequence structure is used as the fragment sequence input for a semi-Markov conditional random field model, wherein each sequence element in the fragment sequence input contains the fragment start and end boundaries of the corresponding field fragment and its compressed feature representation.

[0012] Optionally, the generation of the mapping between the field label set and the field fragment includes: Based on the fragment sequence input, read the fragment-level observations arranged in an ordered sequence of field fragments from the fragment sequence input, where each fragment-level observation contains the start and end boundaries of the corresponding field fragment and its compressed feature representation; After reading fragment-level observations, a field label set is set, and a label identifier is assigned to each field label in the field label set. At the same time, the allowed transition relationships between field labels in the field label set are determined. A semi-Markov conditional random field model is constructed based on fragment-level observations and field label sets. The field label set is set as the state set, the fragment-level observations are set as the observation sequence, and the character position range covered by the field fragment is determined according to the fragment start and end boundaries in the fragment-level observations. The duration of the field fragment is determined by the character position range covered by the field fragment, and the duration of the field fragment is written into the model structure as a state duration constraint. Based on the compressed feature representation in the fragment-level observations, an observation-related feature function is constructed, and based on the adjacent field labels and corresponding durations under the allowed transition relationship, a state transition-related feature function is constructed to calculate the path score of the field label annotation path; Based on the input fragment sequence and the corresponding field label annotation information, supervised annotation information for model training is constructed. Based on the supervision and annotation information, a training objective function is constructed, and the model parameters corresponding to the observation-related feature function and the state transition-related feature function are optimized and updated to obtain the parameters of the trained semi-Markov conditional random field model. Based on the parameters of the trained semi-Markov conditional random field model, the fragment sequence is input into the trained semi-Markov conditional random field model. Based on the model parameters and state persistence constraints, fragment-level joint decoding is performed on the fragment sequence input. The fragment-level joint decoding includes recursively calculating the cumulative path score of each field label under each persistence length according to the arrangement order of fragment-level observations, and recording the corresponding optimal predecessor field label during the recursive calculation process. The optimal predecessor field label of the record is used to perform backtracking to obtain the field label sequence, and the correspondence between the field label set and the field fragment is generated according to the correspondence between the field label sequence and the fragment-level observation.

[0013] Optionally, the generation of structured archival data results includes: Based on the correspondence between the field label set and the field fragment, the field fragment corresponding to each field label is read, and the order in which the field fragments appear in the text sequence to be processed is maintained; For each field tag corresponding to a field fragment, the corresponding field fragment content is extracted from the text sequence to be processed according to the start and end boundaries of the field fragment, forming a set of field fragment content corresponding to the field tag; Based on the set of field fragment content, the field fragment content belonging to the same field tag is concatenated in the order in the set of field fragment content to generate the structured field value corresponding to the field tag; Based on the set of field labels, a mapping relationship is established between each field label and its corresponding structured field value. The mapping relationship between all field labels and structured field values ​​is then summarized to generate structured archive data results.

[0014] The beneficial effects of this invention are: (1) By introducing the concept of fragment-level modeling, this invention integrates the generation of field fragments, feature extraction and field boundary determination in the archival text into the same processing flow, and uses the minimum description length criterion to jointly evaluate the combination of fragment boundaries and multi-granularity features, thereby achieving synchronous optimization of field boundaries and feature representation. This avoids the error accumulation problem caused by the independent field segmentation and feature construction in the prior art, and effectively improves the overall consistency and integrity of the archival data structuring results.

[0015] (2) The present invention constructs a semi-Markov conditional random field model based on compressed fragment-level features, takes field fragments as the whole modeling unit, and performs fragment-level joint decoding on variable-length fields, so that the model can directly output the correspondence between field labels and field fragments. Compared with the traditional sequence labeling method based on characters or fixed windows, it is more suitable for handling complex structures in archive texts with variable field lengths, spanning lines or segments, and improves the accuracy and stability of field recognition.

[0016] (3) This invention introduces noise masking and joint coding mechanism into the minimum description length criterion to effectively distinguish and suppress noise characters and invalid features in the archive text. In the structured result generation stage, it automatically generates structured field values ​​based on the correspondence between field labels and field fragments, which significantly reduces the dependence on manual rules, template configuration and manual verification. This makes the method still have good adaptability and practicality when facing complex scenarios such as different archive types, format changes and optical recognition errors. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the machine learning-based structured extraction method for archival data proposed in this invention; Figure 2 This is a schematic diagram of the joint encoding and evaluation process of fragment boundaries and multi-granularity features based on the minimum description length criterion in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-2 The method for structured extraction of archival data based on machine learning includes the following steps: Obtain the archival text data, and perform character normalization, noise character suppression, and serialization segmentation on the archival text data to obtain the text sequence to be processed; A candidate field fragment set is generated based on the text sequence to be processed. The candidate field fragment set consists of multiple candidate field fragments covering a preset length range, and each candidate field fragment has a fragment start and end boundary. For each candidate field fragment in the candidate field fragment set, a multi-granularity feature set is extracted, which includes character-level features, position-level features, and contextual relationship features; Using the minimum description length criterion as the unified objective function, the fragment boundary combination formed by the start and end boundaries of each candidate field fragment is jointly encoded and evaluated with the corresponding multi-granularity feature set, outputting the fragment partitioning result with the minimum description length, and generating a compressed feature representation for each field fragment in the fragment partitioning result; Based on the field fragments and compressed feature representations in the fragment partitioning results, a fragment sequence input for a semi-Markov conditional random field model is constructed, and a set of field labels is defined to form a fragment-level sequence labeling task. The model parameters are obtained by training a semi-Markov conditional random field model, and fragment-level joint decoding is performed on the fragment sequence input to output the correspondence between the field label set and the field fragment; Based on the correspondence between the field label set and the field fragment, the field fragment is mapped to structured field values ​​to generate structured archive data results.

[0020] In this embodiment, the generation of the text sequence to be processed includes: acquiring archival text data; performing character normalization processing on the archival text data character by character to form a normalized character sequence; the character normalization processing is used to uniformly map full-width characters and half-width characters, uppercase and lowercase characters, traditional characters and simplified characters, homographs with different codes, and preset variant characters to normalized characters; performing noise character suppression processing on the normalized character sequence to form a denoised character sequence; the noise character suppression processing is used to identify and delete non-semantic characters generated by optical character recognition errors, scanning interference, or encoding anomalies, and to merge consecutive blank characters and invalid symbols; performing serialization segmentation processing on the denoised character sequence; determining the initial segmentation position set based on line breaks, page breaks, tabs, and punctuation separators; adjusting the initial segmentation position set in combination with the continuity constraints of number strings, date strings, and number strings; and organizing the denoised character sequence into the text sequence to be processed in the original order based on the adjusted segmentation position set. A newline character is a control character used to indicate the end of a line of text; a page break is a control character used to indicate the end of a page; a tab is a control character used to indicate fixed-interval alignment; and punctuation separators are symbolic characters used to separate semantic units, including commas, periods, semicolons, colons, and their equivalents. The continuity constraint of a number string means that adjacent number characters are treated as a whole as a single numerical unit when they are not separated by a delimiter. The continuity constraint of a date string means that the time expression composed of numbers and date connectors is kept as a whole and not split. The continuity constraint of a number string means that the number identifier composed of numbers, letters and connectors is treated as a continuous sequence. Adjustment refers to removing the split positions within the continuity constraint range from the initial set of split positions, and retaining the split positions that do not violate the continuity constraint, thereby correcting the final set of split positions. Original order organization refers to arranging a sequence of characters according to their order of appearance in the archival text data, without changing the relative positions of the characters, in order to maintain the original arrangement order of the text content.

[0021] In this embodiment, the generation of the candidate field fragment set includes: Based on the text sequence to be processed, character position identifiers are established. Each character in the text sequence to be processed is assigned a continuous position identifier according to the order of its appearance in the text sequence to be processed, and a character sequence with position identifiers is obtained. Establishing character position identifiers refers to assigning a unique sequential number to each character in the text sequence to be processed, according to the order in which the characters appear, to identify the position of the character in the text sequence and to maintain the correspondence between the number and the character order; Assigning consecutive position identifiers means assigning numbers to each character sequentially from the starting number, according to the order of characters in the text sequence to be processed, so that the numbers corresponding to adjacent characters are adjacent and uninterrupted. Based on the character sequence with position identifier, a preset length range for candidate field segments is set. The preset length range is limited by a minimum length value and a maximum length value. The minimum length value limits the candidate field segment to contain one character, and the maximum length value limits the number of characters contained in the candidate field segment to not exceed a preset upper limit. The minimum length of the preset length range Maximum length limited; The preset upper limit refers to the maximum number of characters allowed in advance when generating candidate field fragments. It is used to limit the length of a single candidate field fragment from exceeding this number, thereby constraining the size of the candidate field fragments. Based on a preset length range, any position identifier in the character sequence with position identifier is used as the starting position of the segment. Under the condition that the preset length range is met and the end of the character sequence with position identifier is not exceeded, characters are continuously selected from the starting position of the segment to determine the ending position of the segment, generating a candidate field segment consisting of the character corresponding to the starting position of the segment to the character corresponding to the ending position of the segment. The end of a character sequence with a position identifier refers to the position identifier corresponding to the last character in the character sequence with a position identifier, which is used to indicate the end position of the character sequence; After generating candidate field fragments, record the start and end boundaries of the fragments. The start and end boundaries of the fragments are composed of the start position and the end position of the fragments, and establish a correspondence between the start and end boundaries of the fragments and the content of the candidate field fragments. The content of a candidate field fragment refers to the text content consisting of all consecutive characters from the start position to the end position of the candidate field fragment. Repeatedly perform the following steps for determining the start position of a segment, determining the end position of a segment, generating candidate field segments, and recording the start and end boundaries of segments for all position identifiers in a character sequence with position identifiers. This process is then used to obtain all candidate field segments and their start and end boundaries, forming a set of candidate field segments.

[0022] In this embodiment, the extraction of the multi-granularity feature set includes: for each candidate field segment in the candidate field segment set, reading the start and end boundaries of the segment corresponding to the candidate field segment and determining the character sequence inside the segment and its left and right adjacent context windows; extracting character-level features based on the character sequence inside the segment, including character category features, character normalized morphology features, prefix and suffix substring features of the character sequence, and punctuation distribution features inside the segment; extracting position-level features based on the position of the segment's start and end boundaries in the character sequence with position identifiers, including the relative inline position of the segment's start position and end position, whether it crosses lines, segment length range identifiers, and distance identifiers between the segment and the nearest separator; extracting context relationship features based on the left and right adjacent context windows, including the character sequence features adjacent to the left side of the segment, the character sequence features adjacent to the right side of the segment, the combination features of the segment and adjacent separator types, and the boundary relative relationship features between the segment and adjacent candidate field segments; and indexing and encoding the character-level features, position-level features, and context relationship features according to a preset feature template to obtain the multi-granularity feature set of the candidate field segment. Determining the internal character sequence of a fragment and its left and right adjacent context windows refers to extracting continuous characters from the text sequence to be processed from the start position to the end position of the fragment based on the start and end boundaries of the candidate field fragment as the internal character sequence of the fragment, and selecting adjacent characters on both sides of the start and end boundaries of the fragment forward and backward according to the preset window length to form the left context character sequence and the right context character sequence, respectively. Indexing and encoding according to a preset feature template refers to converting various extracted features into corresponding discrete indices or numbers based on a predefined feature type order and value mapping rules, and organizing them into feature representations according to the order.

[0023] In this embodiment, the generation of compressed feature representation includes: Based on the candidate field fragment set, a fragment boundary combination set is constructed. The fragment boundary combination set consists of multiple fragment boundary combinations. Each fragment boundary combination is composed of the start and end boundaries of the candidate field fragments and corresponds to a set of field fragments that are arranged in order and do not overlap in the text sequence to be processed. Sequentially arranged and non-overlapping field segments refer to field segments arranged in the text sequence to be processed according to the order of their starting positions, and the character position ranges corresponding to any two field segments do not overlap or intersect. For each fragment boundary combination in the fragment boundary combination set, read the corresponding candidate field fragments in the candidate field fragment set for each field fragment in the fragment boundary combination, and perform indexed encoding on the multi-granularity feature set of each field fragment according to the preset feature template to form a fragment-level feature sequence corresponding to the fragment boundary combination. Executing indexed encoding according to a preset feature template means converting each feature value in a multi-granularity feature set into a corresponding discrete index according to a predetermined feature type order and value mapping rules, and organizing them into a feature encoding sequence in the order stated. Based on the fragment-level feature sequence, the start and end boundaries of each field fragment in the fragment boundary combination are used to determine the range of character positions covered by the field fragment. Then, noise mask inference is performed on the characters within the range of the field fragment and their feature dimensions in the fragment-level feature sequence to obtain the noise mask. Based on the noise mask, the fragment-level feature sequence is divided into content feature sequence and noise feature sequence. The range of character positions covered by a field fragment refers to the consecutive character position interval in the text sequence to be processed, defined by the start and end positions of the field fragment. Noise mask inference refers to determining whether each character and its features belong to noise based on the character position range covered by the field fragment, and marking the characters or features that are determined to be noise with noise labels to distinguish noise from non-noise content; Based on content feature sequences and noise feature sequences, a joint coding structure is constructed and mapped. The joint coding structure includes a content coding structure and a noise coding structure. The content coding structure is used to establish a mapping between content feature values ​​and content coding symbols, and the noise coding structure is used to establish a mapping between noise feature values ​​and noise coding symbols. The mapping encodes the content feature sequence into a content coding sequence and the noise feature sequence into a noise coding sequence. Based on the content coding sequence and the noise coding sequence, the total description length corresponding to the segment boundary combination and the local description length of each field segment are calculated. The total description length is obtained by summing the local description lengths of each field segment in the segment boundary combination. The local description length of each field segment is composed of the description length of the content coding sequence and the description length of the noise coding sequence corresponding to the field segment. A dynamic programming solution structure is constructed based on the local description length. The character position identifiers in the text sequence to be processed are used as dynamic programming nodes, candidate field fragments are used as transitions between nodes, and the local description length of the corresponding field fragment is used as the transition cost. Through dynamic programming, the transition path with the minimum total description length is solved among all legal transition paths corresponding to the fragment boundary combination set, and the fragment boundary combination with the minimum total description length is generated as the fragment partitioning result. Constructing a dynamic programming solution structure refers to using the character positions of the text sequence to be processed as state nodes, the start and end boundaries of the candidate field fragments as reachable transitions between nodes, and the local description length of the corresponding field fragments as the transition cost, forming a state transition structure for accumulating costs and solving for the path with the minimum total cost. Dynamic programming is used to find the transition path with the minimum total description length among all legal transition paths corresponding to the fragment boundary combination set. This means that the character position in the text sequence to be processed is used as the stage order, and the legal transition is formed from the end position of the previous field fragment to the start position of the next field fragment. The local description length of the corresponding field fragment is gradually accumulated, and the path with the minimum cumulative description length is selected as the result among all transition paths that satisfy the order and non-overlapping constraints. Output the segmentation result and the content encoding sequence and noise encoding sequence corresponding to each field segment in the segmentation result, and form the compressed feature representation of each field segment in the segmentation result based on the content encoding sequence and noise encoding sequence.

[0024] In this embodiment, the construction of the fragment sequence input includes: Based on the segmentation results, the start and end boundaries of each field segment in the segmentation results are read, and the field segments are sorted according to the order of the start position of the field segments in the text sequence to be processed, forming an ordered sequence of field segments; For each field segment in the ordered sequence of field segments, read the compressed feature representation corresponding to the field segment, and bind the compressed feature representation to the start and end boundaries of the field segment to form a segment-level observation item with the field segment as the basic unit; Based on fragment-level observations, the fragment-level observations are arranged sequentially according to the ordered sequence of field fragments to construct a fragment sequence structure; The ordered sequence of field fragments refers to arranging each field fragment in the text sequence in ascending order of its starting position, so that the order of the field fragments in the sequence is consistent with the order of their appearance in the original text. The fragment sequence structure is used as the fragment sequence input for a semi-Markov conditional random field model, wherein each sequence element in the fragment sequence input contains the fragment start and end boundaries of the corresponding field fragment and its compressed feature representation.

[0025] In this embodiment, the generation of the correspondence between the field label set and the field fragment includes: Based on the fragment sequence input, read the fragment-level observations arranged in an ordered sequence of field fragments from the fragment sequence input, where each fragment-level observation contains the start and end boundaries of the corresponding field fragment and its compressed feature representation; After reading fragment-level observations, a field label set is set, and a label identifier is assigned to each field label in the field label set. At the same time, the allowed transition relationships between field labels in the field label set are determined. A semi-Markov conditional random field model is constructed based on fragment-level observations and field label sets. The field label set is set as the state set, the fragment-level observations are set as the observation sequence, and the character position range covered by the field fragment is determined according to the fragment start and end boundaries in the fragment-level observations. The duration of the field fragment is determined by the character position range covered by the field fragment, and the duration of the field fragment is written into the model structure as a state duration constraint. Determining the duration of a field segment refers to calculating the number of character positions covered by the field segment in the text sequence to be processed based on the start and end boundaries of the field segment, and using this number as the duration of the field segment; Based on the compressed feature representation in the fragment-level observations, an observation-related feature function is constructed, and based on the adjacent field labels and corresponding durations under the allowed transition relationship, a state transition-related feature function is constructed to calculate the path score of the field label annotation path; Based on the input fragment sequence and the corresponding field label annotation information, supervised annotation information for model training is constructed. Based on the supervision and annotation information, a training objective function is constructed, and the model parameters corresponding to the observation-related feature function and the state transition-related feature function are optimized and updated to obtain the parameters of the trained semi-Markov conditional random field model. Based on the parameters of the trained semi-Markov conditional random field model, the fragment sequence is input into the trained semi-Markov conditional random field model. Based on the model parameters and state persistence constraints, fragment-level joint decoding is performed on the fragment sequence input. The fragment-level joint decoding includes recursively calculating the cumulative path score of each field label under each persistence length according to the arrangement order of fragment-level observations, and recording the corresponding optimal predecessor field label during the recursive calculation process. The optimal predecessor field label of the record is used to perform backtracking to obtain the field label sequence, and the correspondence between the field label set and the field fragment is generated according to the correspondence between the field label sequence and the fragment-level observation. Execution backtracking refers to, after completing the recursive calculation, starting from the final fragment position, and gradually restoring the corresponding field label annotation path in reverse order based on the best predecessor field label of the record; Based on the correspondence between the field label sequence and the fragment-level observations, the field labels in the field label sequence are aligned with the order of the fragment-level observations in the fragment sequence input according to their positions in the sequence.

[0026] In this embodiment, the generation of structured archive data results includes: Based on the correspondence between the field label set and the field fragment, the field fragment corresponding to each field label is read, and the order in which the field fragments appear in the text sequence to be processed is maintained; For each field tag corresponding to a field fragment, the corresponding field fragment content is extracted from the text sequence to be processed according to the start and end boundaries of the field fragment, forming a set of field fragment content corresponding to the field tag; Based on the set of field fragment content, the field fragment content belonging to the same field tag is concatenated in the order in the set of field fragment content to generate the structured field value corresponding to the field tag; Field fragment content belonging to the same field label refers to the text content corresponding to multiple field fragments that are assigned the same field label in the correspondence between the field label set and the field fragment; Concatenating the field fragments in the order they appear in the collection refers to connecting the field fragments sequentially to form continuous text, without changing their relative order. Based on the set of field labels, a mapping relationship is established between each field label and its corresponding structured field value. The mapping relationship between all field labels and structured field values ​​is summarized to generate structured archive data results. Establishing a mapping relationship between each field label and its corresponding structured field value means associating each field label with its generated structured field value, thus forming a correspondence between field labels and structured field values. Example

[0027] To verify the feasibility of this invention in practice, it was applied to a large-scale comprehensive archives management institution. In this scenario, the institution has long preserved a large number of historical paper archives and early electronic archives, covering various types such as personnel files, administrative approval materials, contracts, meeting minutes, and business correspondence. These archives differ significantly in their creation period, compilation standards, and layout, including both highly standardized tabular texts and a large amount of freely written narrative text. With increasing demands for archive utilization, relying solely on manual review or data entry is insufficient to meet operational needs. Management personnel wish to automate the conversion of key information from the archives into structured data to support rapid retrieval, statistical analysis, and intelligent association.

[0028] Under current technological conditions, the organization attempted to use a rule-based template approach for structuring archival data. This involved professionals designing field templates and extraction rules for different types of archives in advance, such as extracting fields like name, number, date, and organization name through regular expressions or keyword matching. However, in practical applications, this method proved unstable when faced with changes in archive format, adjustments to field order, cross-line content, or mixed explanatory text. The cost of rule maintenance continued to rise. Statistics showed that after extracting approximately 50,000 archival texts, the field extraction accuracy rate using the rule template approach was only about 72%, with field boundary errors and missing fields accounting for more than 30%. The workload of manual proofreading remained enormous.

[0029] The organization also introduced an automatic extraction scheme based on traditional sequence labeling models, labeling archival text at the character or word level. However, due to the large variation in field lengths within archival texts, and the fact that fields often consist of multiple consecutive characters, the model is prone to generating inconsistent labels within fields when labeling them character by character, leading to incorrect splitting or truncation of fields. Especially in the text generated after scanning archives undergoes optical recognition, there are numerous typos, disordered symbols, and blank lines, making the model sensitive to noise and resulting in insufficient stability of the structured results.

[0030] To address the aforementioned issues, the institution introduced and deployed the machine learning-based structured data extraction method for archives proposed in this invention into its actual system. The system first connects to the archive digitization platform, receiving the archive text data after optical recognition. Once the text enters the system, it automatically performs character normalization processing, converting full-width characters to half-width characters, unifying special symbols from different sources, and suppressing obvious recognition noise characters. Simultaneously, the system performs serialization segmentation of the text based on information such as line breaks, page breaks, and punctuation, forming a text sequence to be processed.

[0031] Building upon this foundation, the system does not directly rely on predefined field templates but automatically generates candidate field fragments from the text sequence to be processed. By assigning continuous positional identifiers to each character in the text and enumerating the start and end boundaries of fragments within a preset length range, the system constructs a set of candidate field fragments covering the entire text. Each candidate field fragment records its start and end positions in the original text, providing a basis for subsequent boundary analysis. This approach allows the system to consider the possibility of both short and long fields simultaneously, rather than being limited to fixed lengths or fixed positions.

[0032] For the generated candidate field fragments, the system further extracts multi-granularity features, including character composition, the fragment's positional relationship in the text, and its association with context characters. Subsequently, an improved minimum description length criterion is introduced to jointly evaluate the boundary combinations and multi-granularity features of the candidate field fragments. During the evaluation process, the system not only calculates the overall description cost of different fragment partitioning schemes but also distinguishes potentially noisy characters and features through a noise masking mechanism, preventing noise information from excessively interfering with structural judgment. Through dynamic programming, the system selects the fragment boundary combination with the minimum description length globally, obtaining the optimal fragment partitioning result, and generates a compressed feature representation for each field fragment.

[0033] After obtaining the segmentation results, the system uses field segments as the overall modeling unit, constructs a segment sequence input, and introduces a semi-Markov conditional random field model to model the segment sequence. Unlike the traditional character-by-character model, this model directly uses field segments as the state persistence unit, which can naturally handle the problem of field lengths not being fixed. The system uses existing manually compiled archives as supervised samples and trains the model based on the correspondence between field labels and field segments. During training, the model learns the transition relationships between fields and the matching patterns between field segments and labels through segment-level joint decoding, gradually forming stable model parameters.

[0034] After model training is complete, the system performs automatic structuring processing on newly input archival text. After the model outputs a sequence of field labels, the system extracts the content of the field fragments from the original text based on the correspondence between the field label set and the field fragments, and then aggregates and concatenates them according to the field labels to finally generate structured archival data. The generated structured data is stored in the form of field key-value pairs and can be directly used in the retrieval and statistics modules of the archival management system.

[0035] In actual operation, the system automatically processed approximately 100,000 documents of different types. Statistical results show that, without manual rule configuration, the overall field extraction accuracy reached over 89%, with the accuracy rate for common core fields exceeding 92%. Compared to the original rule-template method, the accuracy rate improved by approximately 17 percentage points, and the field boundary error rate decreased by over 40%. Regarding processing efficiency, the average processing time for a single document text decreased from approximately 1.8 seconds using the original rule-based method to approximately 0.6 seconds, resulting in a significant improvement in overall processing efficiency. In terms of human involvement, the proportion of post-processing manual proofreading decreased from approximately 35% to less than 10%, significantly reducing the labor costs of document processing.

[0036] Furthermore, the method of this invention exhibits good robustness when dealing with archival texts containing significant optical recognition noise. When tested on scanned archives with numerous typos and poor layout, the stability of the structured results is significantly better than traditional methods, with a marked reduction in field loss rate and erroneous merging. Archival management personnel have reported that the structured results output by the system meet practical business requirements in terms of logical consistency and usability.

[0037] As can be seen from the above embodiments, the machine learning-based structured extraction method for archival data proposed in this invention can effectively solve the problems of low efficiency, insufficient accuracy, and poor adaptability in the existing technology for structured processing of archival text.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for structured extraction of archival data based on machine learning, characterized in that, include: Obtain the archival text data, and perform character normalization, noise character suppression, and serialization segmentation on the archival text data to obtain the text sequence to be processed; A candidate field fragment set is generated based on the text sequence to be processed. The candidate field fragment set consists of multiple candidate field fragments covering a preset length range, and each candidate field fragment has a fragment start and end boundary. For each candidate field fragment in the candidate field fragment set, extract a multi-granularity feature set; Using the minimum description length criterion as the unified objective function, the fragment boundary combination formed by the start and end boundaries of each candidate field fragment is jointly encoded and evaluated with the corresponding multi-granularity feature set, outputting the fragment partitioning result with the minimum description length, and generating a compressed feature representation for each field fragment in the fragment partitioning result; Based on the field fragments and compressed feature representations in the fragment partitioning results, a fragment sequence input for a semi-Markov conditional random field model is constructed, and a set of field labels is defined to form a fragment-level sequence labeling task. The model parameters are obtained by training a semi-Markov conditional random field model, and fragment-level joint decoding is performed on the fragment sequence input to output the correspondence between the field label set and the field fragment; Based on the correspondence between the field label set and the field fragment, the field fragment is mapped to structured field values ​​to generate structured archive data results.

2. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The generation of the text sequence to be processed includes: acquiring archival text data, performing character normalization processing on the archival text data character by character to form a normalized character sequence, performing noise character suppression processing on the normalized character sequence to form a denoised character sequence, performing serialization segmentation processing on the denoised character sequence, determining the initial segmentation position set based on line breaks, page breaks, tabs, and punctuation separators, adjusting the initial segmentation position set in combination with the continuity constraints of number strings, date strings, and number strings, and organizing the denoised character sequence into the text sequence to be processed based on the adjusted segmentation position set in the original order.

3. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The generation of the candidate field fragment set includes: Based on the text sequence to be processed, character position identifiers are established. Each character in the text sequence to be processed is assigned a continuous position identifier according to the order of its appearance in the text sequence to be processed, and a character sequence with position identifiers is obtained. Based on the character sequence with position identifier, a preset length range for candidate field segments is set. The preset length range is limited by a minimum length value and a maximum length value. The minimum length value limits the candidate field segment to contain one character, and the maximum length value limits the number of characters contained in the candidate field segment to not exceed a preset upper limit. Based on a preset length range, any position identifier in the character sequence with position identifier is used as the starting position of the segment. Under the condition that the preset length range is met and the end of the character sequence with position identifier is not exceeded, characters are continuously selected from the starting position of the segment to determine the ending position of the segment, generating a candidate field segment consisting of the character corresponding to the starting position of the segment to the character corresponding to the ending position of the segment. After generating candidate field fragments, record the start and end boundaries of the fragments. The start and end boundaries of the fragments are composed of the start position and the end position of the fragments, and establish a correspondence between the start and end boundaries of the fragments and the content of the candidate field fragments. Repeatedly perform the following steps for determining the start position of a segment, determining the end position of a segment, generating candidate field segments, and recording the start and end boundaries of segments for all position identifiers in a character sequence with position identifiers. This process is then used to obtain all candidate field segments and their start and end boundaries, forming a set of candidate field segments.

4. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The extraction of multi-granularity feature sets includes: for each candidate field fragment in the candidate field fragment set, reading the start and end boundaries of the fragment corresponding to the candidate field fragment and determining the character sequence inside the fragment and its left and right adjacent context windows; extracting character-level features based on the character sequence inside the fragment; extracting position-level features based on the position of the fragment start and end boundaries in the character sequence with position identifiers; extracting context relationship features based on the left and right adjacent context windows; and indexing and encoding the character-level features, position-level features and context relationship features according to a preset feature template to obtain the multi-granularity feature set of the candidate field fragment.

5. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The improved minimum description length criteria include: Based on the candidate field fragment set, a fragment boundary combination set is constructed. The fragment boundary combination set consists of multiple fragment boundary combinations. Each fragment boundary combination is composed of the start and end boundaries of the candidate field fragments and corresponds to a set of field fragments that are arranged in order and do not overlap in the text sequence to be processed. For each fragment boundary combination in the fragment boundary combination set, read the corresponding candidate field fragments in the candidate field fragment set for each field fragment in the fragment boundary combination, and perform indexed encoding on the multi-granularity feature set of each field fragment according to the preset feature template to form a fragment-level feature sequence corresponding to the fragment boundary combination. Based on the fragment-level feature sequence, the start and end boundaries of each field fragment in the fragment boundary combination are used to determine the range of character positions covered by the field fragment. Then, noise mask inference is performed on the characters within the range of the field fragment and their feature dimensions in the fragment-level feature sequence to obtain the noise mask. Based on the noise mask, the fragment-level feature sequence is divided into content feature sequence and noise feature sequence. Based on content feature sequences and noise feature sequences, a joint coding structure is constructed and mapped. The joint coding structure includes a content coding structure and a noise coding structure. The mapping encodes the content feature sequences into content coding sequences and the noise feature sequences into noise coding sequences. Based on the content coding sequence and the noise coding sequence, calculate the total description length corresponding to the fragment boundary combination and the local description length of each field fragment; A dynamic programming solution structure is constructed based on the local description length. The character position identifiers in the text sequence to be processed are used as dynamic programming nodes, candidate field fragments are used as transitions between nodes, and the local description length of the corresponding field fragment is used as the transition cost. Through dynamic programming, the transition path with the minimum total description length is solved among all legal transition paths corresponding to the fragment boundary combination set, and the fragment boundary combination with the minimum total description length is generated as the fragment partitioning result. Output the segmentation result and the content encoding sequence and noise encoding sequence corresponding to each field segment in the segmentation result, and form the compressed feature representation of each field segment in the segmentation result based on the content encoding sequence and noise encoding sequence.

6. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The construction of the fragment sequence input includes: Based on the segmentation results, the start and end boundaries of each field segment in the segmentation results are read, and the field segments are sorted according to the order of the start position of the field segments in the text sequence to be processed, forming an ordered sequence of field segments; For each field segment in the ordered sequence of field segments, read the compressed feature representation corresponding to the field segment, and bind the compressed feature representation to the start and end boundaries of the field segment to form a segment-level observation item with the field segment as the basic unit; Based on fragment-level observations, the fragment-level observations are arranged sequentially according to the ordered sequence of field fragments to construct a fragment sequence structure; The fragment sequence structure is used as the fragment sequence input for a semi-Markov conditional random field model, wherein each sequence element in the fragment sequence input contains the fragment start and end boundaries of the corresponding field fragment and its compressed feature representation.

7. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The generation of the correspondence between the field label set and the field fragment includes: Based on the fragment sequence input, read the fragment-level observations arranged in an ordered sequence of field fragments from the fragment sequence input, where each fragment-level observation contains the start and end boundaries of the corresponding field fragment and its compressed feature representation; After reading fragment-level observations, a field label set is set, and a label identifier is assigned to each field label in the field label set. At the same time, the allowed transition relationships between field labels in the field label set are determined. A semi-Markov conditional random field model is constructed based on fragment-level observations and field label sets. The field label set is set as the state set, the fragment-level observations are set as the observation sequence, and the character position range covered by the field fragment is determined according to the fragment start and end boundaries in the fragment-level observations. The duration of the field fragment is determined by the character position range covered by the field fragment, and the duration of the field fragment is written into the model structure as a state duration constraint. Observation-related feature functions are constructed based on the compressed feature representation in fragment-level observations, and state transition-related feature functions are constructed based on adjacent field labels and corresponding durations under allowed transition relations. Based on the input fragment sequence and the corresponding field label annotation information, supervised annotation information for model training is constructed. Based on the supervision and annotation information, a training objective function is constructed, and the model parameters corresponding to the observation-related feature function and the state transition-related feature function are optimized and updated to obtain the parameters of the trained semi-Markov conditional random field model. Based on the parameters of the trained semi-Markov conditional random field model, the fragment sequence is input into the trained semi-Markov conditional random field model. Based on the model parameters and state persistence constraints, fragment-level joint decoding is performed on the fragment sequence input. The fragment-level joint decoding includes recursively calculating the cumulative path score of each field label under each persistence length according to the arrangement order of fragment-level observations, and recording the corresponding optimal predecessor field label during the recursive calculation process. The optimal predecessor field label of the record is used to perform backtracking to obtain the field label sequence, and the correspondence between the field label set and the field fragment is generated according to the correspondence between the field label sequence and the fragment-level observation.

8. The method for structured extraction of archival data based on machine learning according to claim 1, characterized in that, The generation of structured archival data results includes: Based on the correspondence between the field label set and the field fragment, the field fragment corresponding to each field label is read, and the order in which the field fragments appear in the text sequence to be processed is maintained; For each field tag corresponding to a field fragment, the corresponding field fragment content is extracted from the text sequence to be processed according to the start and end boundaries of the field fragment, forming a set of field fragment content corresponding to the field tag; Based on the set of field fragment content, the field fragment content belonging to the same field tag is concatenated in the order in the set of field fragment content to generate the structured field value corresponding to the field tag; Based on the set of field labels, a mapping relationship is established between each field label and its corresponding structured field value. The mapping relationship between all field labels and structured field values ​​is then summarized to generate structured archive data results.

Images