A crf enhancement based bert address level element classification method and system

Abstract

The application relates to the technical field of natural language processing, in particular to a BERT address hierarchical element classification method and system based on CRF enhancement, which comprises the following steps: acquiring an address text sample, marking address characters in the address text sample according to a BIO label system to obtain labeled training data; constructing model input data according to the labeled training data; constructing an address hierarchical element classification model, wherein the address hierarchical element classification model comprises a BERT pre-training encoding layer, a linear classification layer and a CRF layer; training the address hierarchical element classification model by using the model input data; inputting to-be-analyzed address text into the trained address hierarchical element classification model, decoding a label sequence of the to-be-analyzed address text, and obtaining an address hierarchical element label sequence corresponding to the to-be-analyzed address text; and merging address characters in the to-be-analyzed address text according to boundary marks and address element categories in the address hierarchical element label sequence to obtain a structured address hierarchical element.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a BERT address hierarchy feature classification method and system based on CRF enhancement. Background Technology

[0002] Address resolution is a crucial task in natural language processing and geographic information processing. It's typically used to break down unstructured address text from user input, logistics orders, government registrations, business information, and map searches into structured address hierarchical elements. Chinese address text usually contains various elements such as country, provincial-level administrative region, municipal-level administrative region, district / county, development zone, township / street, community, road, house number, point of interest, building number, unit, floor number, room number, and auxiliary locator terms. These elements exhibit clear hierarchical and boundary relationships. Due to the presence of abbreviations, omissions, redundancy, non-standard order, complex point of interest names, and adjacent house and road names in Chinese addresses, accurate identification and boundary delineation of address hierarchical elements are highly challenging.

[0003] Among existing address resolution methods, one type primarily relies on administrative division databases, road databases, place name databases, rule templates, regular expressions, and string similarity matching algorithms to segment, match, and complete address text. This type of method is applicable to address text with a high degree of standardization, but it lacks adaptability to non-standard addresses, mixed addresses, redundant addresses, and addresses with missing hierarchical levels, and the rule maintenance cost is high. When address text simultaneously contains points of interest, buildings, units, room numbers, and auxiliary locator terms, relying solely on rule and dictionary matching can easily lead to problems such as misclassification of address element boundaries, incorrect hierarchical attribution, and omission of final-level address information.

[0004] With the development of deep learning technology, token classification methods based on pre-trained language models have been applied to Chinese address element recognition. These methods typically input characters or tokens from the address text into a BERT model, which extracts contextual semantic features and then outputs corresponding address element labels for each token through a classification layer. Because the BERT model can utilize contextual information, it has stronger semantic expressive power compared to traditional rule-based methods. However, token classification methods using BERT alone with a classification layer usually predict the category of each token relatively independently, without sufficiently constraining the transition relationships between label sequences. For example, the initial label, internal labels, and subsequent address-level labels of an address element in the address text have continuity and order. Without constraints on the overall structure of the label sequence, boundary recognition errors can easily occur at adjacent positions such as district / county and street, road and house number, point of interest and building number, and building number and room number.

[0005] Existing technologies also include schemes that combine BERT, BiLSTM, and CRF for Chinese address element parsing. These typically use BERT for character vectorization, BiLSTM to extract contextual dependencies, and CRF for address element labeling. Other schemes use BERT+CRF for address named entity segmentation and further integrate external databases such as administrative division databases, road databases, point-of-interest databases, and fence databases for multi-level standardized matching and verification. These schemes demonstrate the feasibility of using CRF for sequence labeling tasks. However, in the specific address-level element classification process, a complete design is still needed, focusing on fine-grained hierarchical label construction of address text, model input data construction, handling of special markers and padding positions, valid token position constraints, CRF training loss calculation, and post-decoding address character merging rules. Otherwise, issues such as invalid tokens participating in training, special marker positions interfering with the label sequence, and inconsistencies between label boundaries and address characters can easily affect the stability of address-level element classification results. Summary of the Invention

[0006] This invention aims to solve the above-mentioned technical problems by providing a BERT address-level feature classification method and system based on CRF enhancement. By constructing a key-value memory structure within the Transformer layer and utilizing query update and multi-round memory retrieval mechanisms to complete historical knowledge retrieval, it solves the problems of insufficient historical knowledge activation, limited intra-layer memory retrieval capability, and significant memory interference between tasks in the prior art.

[0007] The objective of this invention can be achieved through the following technical solutions: A CRF-enhanced BERT address hierarchy feature classification method includes the following steps: Obtain address text samples, and label the address characters in the address text samples according to the BIO labeling system to obtain labeled training data. The BIO labeling system includes boundary markers used to represent the boundaries of address elements and address element categories used to represent the address hierarchy type. Model input data is constructed based on labeled training data. The model input data includes address token sequences, label sequences, and valid token masks. The valid token masks are used to limit the positions of address tokens participating in the CRF layer calculation. An address-level feature classification model is constructed, which includes a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer is used to extract the context hidden state of each token in the address token sequence. The linear classification layer is used to generate the emission score of each tag corresponding to each token based on the context hidden state. The CRF layer is used to determine the address-level feature tag sequence based on the emission score, tag transition score, and effective token mask. The address-level feature classification model is trained using the model input data, and the CRF layer calculates the sequence labeling loss based on the emission score, label sequence, and effective token mask. Obtain the address text to be parsed, input the address text to be parsed into the trained address hierarchy feature classification model, and decode the label sequence of the address text to be parsed through the CRF layer to obtain the address hierarchy feature label sequence corresponding to the address text to be parsed. Based on the boundary markers and address element categories in the address hierarchy element label sequence, the address characters in the address text to be parsed are merged to obtain structured address hierarchy elements.

[0008] Furthermore, address characters in the address text samples are labeled according to the BIO tagging system, including: Address character annotations include labels formed by a combination of boundary markers and address element categories; Among them, the boundary identifiers include B identifier, I identifier and O identifier. The B identifier is used to indicate the starting position of the address element, the I identifier is used to indicate the internal position of the address element, and the O identifier is used to indicate the position of the non-address element. Address element categories include country, provincial administrative region, municipal administrative region, district / county, development zone, township / street, community, main road, sub-road, house number, sub-road number, point of interest, sub-point of interest, building number, unit, floor number, room number, business / shop name, auxiliary location keywords, redundant information, and unclassifiable information.

[0009] Furthermore, the labeled training data is stored in a character-by-character annotation format. The character-by-character annotation format includes annotation records corresponding to multiple address text samples. Each annotation record includes an address character and a label corresponding to the address character. The label is formed by a combination of boundary markers and address element categories. Annotation records corresponding to different address text samples are separated by blank lines.

[0010] Furthermore, the model input data is constructed based on the labeled training data, including: The address text samples are tokenized using a pre-trained tokenizer to obtain an address token sequence. Add a classification marker [CLS] to the beginning of the address token sequence and an end marker [SEP] to the end of the address token sequence to obtain the address token sequence with the added markers; The length of the marked address token sequence is uniformly processed according to the preset maximum length. Specifically, when the length of the marked address token sequence is less than the preset maximum length, padding is used to make it full; when the length of the marked address token sequence is greater than the preset maximum length, it is truncated to the preset maximum length. The address token sequence with uniform length is converted into an input ID sequence, and an attention mask is constructed based on the valid token positions and padding marker positions in the address token sequence with uniform length. The labels in the labeled training data are converted into a sequence of label IDs, and the model input data is constructed based on the sequence of label IDs and the attention mask.

[0011] Furthermore, the labels in the labeled training data are converted into a sequence of label IDs, and the model input data is constructed based on the sequence of label IDs and the attention mask, including: Set the labels at the corresponding positions of the category marker, the end marker, and the fill marker to ignore flags; Convert the tag corresponding to the valid token position into a tag ID to obtain the tag ID sequence; The effective token position and the padding mark position are determined based on the attention mask, and the corresponding position of the category mark and the corresponding position of the end mark are determined based on the ignore flag in the tag ID sequence; A valid token mask is constructed based on the valid token position, the padding mark position, the corresponding category mark position, and the corresponding end mark position. The mask value corresponding to the valid token position is the first mask value, and the mask value corresponding to the padding mark position, the corresponding category mark position, and the corresponding end mark position is the second mask value.

[0012] Furthermore, the address-level feature classification model also includes a Dropout regularization layer, which is set between the BERT pre-trained encoding layer and the linear classification layer. The BERT pre-trained encoding layer uses a Chinese pre-trained BERT model. The BERT pre-trained encoding layer receives the input ID sequence and attention mask, and outputs the context hidden state corresponding to each token. The Dropout regularization layer performs random deactivation processing on the context hidden state according to a preset random deactivation probability, and obtains the processed context hidden state. The linear classification layer maps the processed contextual hidden state to the emission score of each token corresponding to each tag; The CRF layer determines the address level element tag sequence based on the transmission score and tag transfer score.

[0013] Furthermore, the address-level feature classification model is trained using the model input data, including: Input the input ID sequence and attention mask into the BERT pre-trained encoding layer to obtain the context hidden state corresponding to each token; The contextual hidden state is input into a linear classification layer to obtain the emission score; Input the emission score, tag ID sequence, and valid token mask into the CRF layer; The CRF layer calculates the sentence-level log-likelihood based on the emission score, tag ID sequence, valid token mask, and tag transition score. The sequence labeling loss is obtained by taking the negative value of the sentence-level log-likelihood and averaging it.

[0014] Furthermore, training the address-level feature classification model using the model input data also includes: The preprocessed address data is encapsulated into a dataset object and a data loader object, and the batch data is pruned according to the effective token length using a batch pruning function; Set training hyperparameters, including maximum training epochs, batch size, learning rate, weight decay, warm-up ratio, gradient accumulation steps, and maximum norm of gradient clipping. The AdamW optimizer is used to update the model parameters of the address hierarchy feature classification model. The model parameters are divided into a weighted decay parameter group and a non-weighted decay parameter group. The learning rate is adjusted using a linear learning rate scheduler with preheating. Iterate through the training data in the dataset object, calculate the sequence label loss for each batch, backpropagate the sequence label loss, and save the address level feature classification model with the minimum loss during training.

[0015] Furthermore, based on the boundary markers and address element categories in the address hierarchy element label sequence, the address characters in the address text to be parsed are merged to obtain structured address hierarchy elements, including: Read the label corresponding to each address character in the address hierarchy element label sequence; When the label corresponding to the current address character contains the 'B' identifier, the current address character is used as the starting character of the current address element, and the address element category corresponding to the current address character is used as the current address element category. When the label corresponding to the next address character contains the 'I' identifier, and the address element category corresponding to the next address character is consistent with the current address element category, the next address character will be merged into the current address element. When the next tag containing the 'B' identifier is read, the merging of the current address element ends, and the address character corresponding to the next tag containing the 'B' identifier is used as the starting character of the next address element. When a tag containing the 'O' identifier is read, the address character corresponding to the 'O' identifier is not incorporated into the structured address hierarchy elements; Output structured address hierarchy elements according to the order in which each address element is arranged in the address text to be parsed.

[0016] A CRF-enhanced BERT address hierarchy feature classification system includes: The labeled data generation module is used to obtain address text samples, and to label the address characters in the address text samples according to the BIO label system to obtain labeled training data. The BIO label system includes boundary markers used to represent the boundaries of address elements and address element categories used to represent the address level type. The model input construction module is used to construct model input data based on labeled training data. The model input data includes an address token sequence, a label sequence, and a valid token mask. The valid token mask is used to limit the location of address tokens participating in the CRF layer calculation. The model building module is used to build an address-level feature classification model. The address-level feature classification model includes a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer is used to extract the context hidden state of each token in the address token sequence. The linear classification layer is used to generate the emission score of each token corresponding to each label based on the context hidden state. The CRF layer is used to determine the address-level feature label sequence based on the emission score, label transition score, and effective token mask. The model training module is used to train the address-level feature classification model using the model input data, enabling the CRF layer to calculate the sequence labeling loss based on the emission score, label sequence, and effective token mask. The label decoding module is used to obtain the address text to be parsed. The address text to be parsed is input into the trained address hierarchical feature classification model. The CRF layer decodes the label sequence of the address text to be parsed to obtain the address hierarchical feature label sequence corresponding to the address text to be parsed. The structured output module is used to merge address characters in the address text to be parsed based on the boundary markers and address element categories in the address hierarchy element label sequence, thereby obtaining structured address hierarchy elements.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention annotates address characters in address text samples using the BIO tagging system and combines boundary markers with address element categories to form address-level element labels. This ensures that training data simultaneously includes address element category information and address element boundary information. Therefore, the model training phase can learn based on the starting position, internal position, and non-address element position of address characters, reducing the unclear boundary problems caused by classifying solely by address category. It is suitable for recognizing multi-level address elements such as provincial-level administrative regions, municipal-level administrative regions, districts / counties, townships / streets, roads, house numbers, points of interest, building numbers, units, floor numbers, and room numbers.

[0018] 2. In the preprocessing stage, this invention constructs an input ID sequence, an attention mask, a label ID sequence, and a valid token mask. The positions corresponding to the classification marker, end marker, and padding marker are set to not participate in the model loss calculation, allowing model training and CRF layer calculations to focus on the valid token positions corresponding to the actual address characters. This reduces the interference of special markers and padding markers on address boundary learning, ensures consistency between the model input, label sequence, and valid character positions, and reduces the risk of inconsistency between training labels and inference output.

[0019] 3. This invention extracts the contextual hidden state of each token through a BERT pre-trained encoding layer, generates emission scores for each token corresponding to each tag through a linear classification layer, and determines the address-level element tag sequence through a CRF layer combined with the tag transition score. During training, the sequence labeling loss is calculated based on the emission score, tag ID sequence, and effective token mask. During inference, the tag sequence is decoded through a CRF layer, and address characters are merged according to boundary markers and address element categories. Therefore, the address tag sequence no longer relies solely on the independent classification results of a single token, reducing tag jumps, order reversals, and cross-level misclassification problems, and converting unstructured address text into structured address-level elements. Attached Figure Description

[0020] Figure 1 A flowchart of the BERT address hierarchy feature classification method based on CRF enhancement provided in this embodiment of the invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0022] like Figure 1 The illustrated BERT address hierarchy feature classification method based on CRF enhancement includes the following steps: Obtain address text samples and label the address characters in the address text samples according to the BIO labeling system to obtain labeled training data. The BIO labeling system includes boundary markers for representing address element boundaries and address element categories for representing address hierarchy types. This allows the training data to carry both address element category information and address element boundary information, providing a supervised data foundation for the subsequent model to learn address hierarchy types and boundary positions.

[0023] Model input data is constructed based on labeled training data. The model input data includes address token sequences, label sequences, and valid token masks. The valid token mask is used to limit the address token positions participating in the CRF layer calculation. The original address text and artificial labels are converted into an input format that the model can read. The influence of special markers and padding markers on the CRF layer training is eliminated by the valid token mask, so that the model training focuses on the real address character positions.

[0024] An address-level feature classification model is constructed, comprising a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer extracts the contextual hidden state of each token in the address token sequence. The linear classification layer generates emission scores for each token's corresponding label based on the contextual hidden state. The CRF layer determines the address-level feature label sequence based on the emission scores, label transition scores, and effective token masks. By utilizing the BERT pre-trained encoding layer to extract the contextual semantic features of the address text, the linear classification layer to generate token-level label emission scores, and the CRF layer to constrain the transition relationships between labels, a classification model is formed that transforms address characters into address-level feature label sequences.

[0025] By training an address-level feature classification model using the model input data, the CRF layer calculates the sequence labeling loss based on the emission score, label sequence, and effective token mask. This allows the BERT pre-trained encoding layer, linear classification layer, and CRF layer to be jointly trained on labeled training data, and the CRF layer to calculate the loss based on the complete label sequence, avoiding the use of only the local classification result of a single token as the training objective.

[0026] Obtain the address text to be parsed, input the address text to be parsed into the trained address hierarchy feature classification model, and decode the label sequence of the address text to be parsed through the CRF layer to obtain the address hierarchy feature label sequence corresponding to the address text to be parsed; through this step, the address text to be parsed is converted into a label sequence with boundary markers and address feature categories, so that each address character in the unstructured address text has a corresponding address hierarchy label.

[0027] Based on the boundary markers and address element categories in the address hierarchy element label sequence, address characters in the address text to be parsed are merged to obtain structured address hierarchy elements. This converts the label sequence output by the CRF layer into a directly usable structured address result, allowing consecutive characters in the address text to be grouped according to address element categories, forming hierarchical address fields such as province, city, district / county, road, house number, point of interest, building number, and room number.

[0028] Specifically, after obtaining the address text sample, the address text sample is split at the character level to obtain the address character sequence. For each address character in the address character sequence, it is labeled character by character according to the BIO tagging system. The BIO tagging system consists of boundary identifiers and address element categories, where the boundary identifiers are used to indicate the position of the address character in the corresponding address element, and the address element category is used to indicate the address level type to which the address character belongs.

[0029] In specific annotation, if an address character is the starting character of an address element, its boundary is marked with a "B" identifier; if an address character is located within the same address element, its boundary is marked with an "I" identifier; if an address character does not belong to the address element to be extracted, its boundary is marked with an "O" identifier. Address element categories are determined according to address hierarchy type, including country, provincial administrative region, municipal administrative region, district / county, development zone, township / street, community, main road, sub-road, house number, sub-road number, point of interest, sub-point of interest, building number, unit, floor number, room number, business / shop name, auxiliary locator terms, redundant information, and unclassifiable information. The technical disclosure document lists corresponding address element categories such as country, prov, city, district, devzone, town, community, road, subroad, roadno, subroadno, poi, subpoi, houseno, cellno, floorno, roomno, person, assistant, redundant, and otherinfo.

[0030] When creating labels, boundary markers are combined with address element categories into a complete label. For example, when the address characters belong to the starting position of a township / street, it is labeled B-town; when the address characters belong to the interior of a township / street, it is labeled I-town; when the address characters belong to the starting position of a point of interest, it is labeled B-poi; when the address characters belong to the interior of a point of interest, it is labeled I-poi; when the address characters belong to the starting position of a building number, it is labeled B-houseno; when the address characters belong to the interior of a building number, it is labeled I-houseno. The example given in the technical disclosure is "Building B, No. 1097, Taihe Community, Longgang Town," where "Long" is labeled B-town, "Gang" and "Town" are labeled I-town, "Tai" is labeled B-poi, "He," "Xiao," and "Community" are labeled I-poi, "B" is labeled B-houseno, "Building" is labeled I-houseno, and "1097" is labeled B-roomno.

[0031] Using the above annotation method, each address character corresponds to a label jointly defined by boundary markers and address element categories, ensuring that the starting position, internal position, non-address element position, and address element category of the same address text are recorded simultaneously. Address characters that cannot be categorized into country, provincial administrative region, municipal administrative region, district / county, development zone, township / street, community, main road, sub-road, house number, sub-road number, point of interest, sub-point of interest, building number, unit, floor number, room number, business / shop name, auxiliary locator terms, or redundant information are labeled as unclassifiable information.

[0032] Specifically, after constructing the labeled training data, the address text samples are processed as model input. First, a pre-trained tokenizer matching the BERT pre-trained encoding layer is invoked to tokenize the address text samples, converting continuous address text into a sequence of address tokens. For Chinese address text, the pre-trained tokenizer can generate tokens at the character level or according to its vocabulary rules. When the address text contains numbers, letters, road numbers, building numbers, room numbers, etc., these elements are converted into corresponding tokens according to the pre-trained tokenizer's segmentation rules.

[0033] Then, a classification marker [CLS] is added to the beginning of the address token sequence, and an end marker [SEP] is added to the end of the address token sequence to obtain the marked address token sequence. For example, when the address text sample is "No. 1097, Building B, Taihe Community, Longgang Town", after tokenization, the corresponding address token sequence is obtained. Then, [CLS] is added before the address token sequence and [SEP] is added after the address token sequence to form an input sequence with sequence start and end markers.

[0034] Furthermore, the length of the marked address token sequence is uniformly processed according to a preset maximum length. The preset maximum length can be pre-set based on the length distribution of the address text samples; in one specific embodiment, the preset maximum length is set to 50. When the length of the marked address token sequence is less than the preset maximum length, a padding marker [PAD] is added to the end of the marked address token sequence until the sequence length reaches the preset maximum length; when the length of the marked address token sequence is greater than the preset maximum length, the first tokens are retained according to the preset maximum length, so that the length of the processed address token sequence is consistent with the preset maximum length.

[0035] Further, the address token sequence with uniform length is input into the vocabulary mapping unit corresponding to the pre-trained word segmenter, mapping each token to a corresponding integer number to obtain the input ID sequence. Simultaneously, an attention mask is constructed based on the address token sequence with uniform length. The attention mask is used to distinguish between valid token positions and padding marker positions. [CLS], [SEP], and the tokens converted from address text samples are all considered valid token positions, while the padding marker [PAD] is considered a padding marker position. In one specific implementation, the attention mask value corresponding to a valid token position is set to 1, and the attention mask value corresponding to a padding marker position is set to 0.

[0036] Furthermore, a mapping relationship is established between address-level feature labels and label IDs, converting the labels in the labeled training data into a sequence of label IDs. The label ID sequence and the address token sequence (after uniform length processing) are positionally correlated, ensuring that each address token corresponds to a single label ID. For labels formed by combining boundary markers and address feature categories, the corresponding label ID is queried according to a pre-defined label mapping table; for example, B-town, I-town, B-poi, I-poi, B-houseno, and I-houseno are mapped to their respective label IDs. Thus, the input ID sequence, attention mask, label ID sequence, and data related to the address token sequence together constitute the model input data.

[0037] Specifically, after obtaining the address token sequence with uniform length processing, attention mask, and labeled training data, the labels in the labeled training data are processed into IDs, and a valid token mask is constructed simultaneously.

[0038] First, for the classification marker [CLS], end marker [SEP], and padding marker [PAD] in the address token sequence after length unification, since these markers do not belong to the actual address characters in the address text sample, the labels at the corresponding positions of the classification marker, end marker, and padding marker are uniformly set to ignore flags. In one specific implementation, the ignore flag is set to -100, so that the above positions are not used as effective supervision labels in loss calculation during model training.

[0039] Next, a mapping relationship is established between tags and tag IDs. Tags are address-level element labels formed by combining boundary identifiers and address element categories, such as B-town, I-town, B-poi, I-poi, B-houseno, I-houseno, and B-roomno. Based on a pre-defined tag mapping table, the tags corresponding to valid token positions are converted into corresponding tag IDs, thus obtaining a tag ID sequence. The tag ID sequence maintains the same length as the address token sequence after length unification, and each position in the tag ID sequence corresponds one-to-one with the corresponding token position in the address token sequence.

[0040] Further, the positions of valid tokens and padding markers are determined based on the attention mask. Specifically, the positions in the attention mask representing valid input correspond to valid token positions, and the positions in the attention mask representing padding input correspond to padding marker positions. In one specific implementation, positions with an attention mask value of 1 are determined as valid token positions, and positions with an attention mask value of 0 are determined as padding marker positions.

[0041] Furthermore, the positions corresponding to the classification marker and the end marker are determined based on the ignore flags in the label ID sequence. Since the classification marker [CLS] and the end marker [SEP] are set as ignore flags, their positions can be identified within the label ID sequence. For the padding marker [PAD], it can be determined simultaneously through the padding marker position in the attention mask and the ignore flags in the label ID sequence.

[0042] After determining the positions of the valid token, padding marker, classification marker, and end marker, a valid token mask is constructed. The valid token mask identifies the range of positions in the CRF layer involved in the calculation. Specifically, the valid token position corresponding to the actual address character in the valid token mask is set as the first mask value, and the positions corresponding to the padding marker, classification marker, and end marker positions in the valid token mask are set as the second mask value. In one specific implementation, the first mask value is 1, and the second mask value is 0.

[0043] Therefore, the model input data includes an input ID sequence, an attention mask, a label ID sequence, and a valid token mask. The input ID sequence represents the numerical input of the address token sequence, the attention mask distinguishes between valid input and padding input, the label ID sequence represents the supervision label corresponding to each token position, and the valid token mask limits the position of the actual address characters for sequence labeling calculation by the CRF layer.

[0044] Specifically, the address-level feature classification model consists of a BERT pre-trained encoding layer, a Dropout regularization layer, a linear classification layer, and a CRF layer connected in sequence. The BERT pre-trained encoding layer uses a Chinese pre-trained BERT model. The input to the Chinese pre-trained BERT model includes an input ID sequence and an attention mask. The input ID sequence represents the address token sequence after uniform length processing, and the attention mask distinguishes between valid token positions and padding marker positions.

[0045] After receiving the input ID sequence and attention mask, the BERT pre-trained encoding layer performs contextual semantic encoding on each token in the address token sequence, outputting the contextual hidden state corresponding to each token. The contextual hidden state is a vector representation containing contextual semantic information. In one specific implementation, the dimension of the contextual hidden state is consistent with the hidden layer dimension of the Chinese pre-trained BERT model, which is 768.

[0046] The Dropout regularization layer is placed between the BERT pre-trained encoding layer and the linear classification layer. The Dropout regularization layer receives the context hidden state and randomly deactivates a portion of the neuron outputs in the context hidden state according to a preset random deactivation probability, resulting in the processed context hidden state. In one specific implementation, the preset random deactivation probability is the hidden_dropout_prob setting in the Chinese pre-trained BERT model configuration, with a preset random deactivation probability of 0.1.

[0047] The linear classification layer receives the processed context hidden state and maps the context hidden state corresponding to each token to the label space, obtaining the emission score for each token and its corresponding label. The input dimension of the linear classification layer is consistent with the dimension of the context hidden state, and the output dimension is consistent with the number of address-level feature labels. The emission score is used to represent the score of the corresponding token being labeled as each address-level feature label.

[0048] The CRF layer receives the transmission score and combines it with the tag transfer score to determine the address-level element tag sequence. The tag transfer score is used to represent the transfer relationship between adjacent tags. The CRF layer scores the tag path corresponding to the complete address token sequence based on the transmission score of each tag corresponding to each token and the tag transfer score between adjacent tags, and determines the address-level element tag sequence whose score meets the preset decoding rules.

[0049] During the forward processing of the model, the address token sequence first passes through the BERT pre-trained encoding layer to obtain the context hidden state, then through the Dropout regularization layer to obtain the processed context hidden state, then through the linear classification layer to obtain the emission score, and finally through the CRF layer combined with the label transition score to output the address level element label sequence.

[0050] Specifically, before training the address-level feature classification model, the preprocessed address data is encapsulated into a dataset object. Each data entry in the dataset object includes an input ID sequence, an attention mask, a label ID sequence, and a valid token mask. The dataset object is further encapsulated into a data loader object, enabling the model input data to be read in batches during training.

[0051] During batch reading, a batch pruning function is used to prune the lengths of the input ID sequence, attention mask, label ID sequence, and valid token mask within the same batch. Specifically, the length of the valid token corresponding to each address text sample in the current batch is first calculated. Then, the maximum length of the valid token in the current batch is used as the pruning length to simultaneously prune the input ID sequence, attention mask, label ID sequence, and valid token mask, ensuring that the sequences within the same batch maintain consistent lengths and removing padding portions at the end of the batch that are not included in the calculation.

[0052] During training, the input ID sequence and attention mask from the current batch are fed into the BERT pre-trained encoding layer. The BERT pre-trained encoding layer determines the vector representation of each token based on the input ID sequence, distinguishes the positions of valid tokens and padding markers based on the attention mask, and outputs the contextual hidden state corresponding to each token. The contextual hidden state serves as the semantic representation of each token within the address text context.

[0053] Then, the context hidden state is input into the linear classification layer. The linear classification layer performs a linear mapping on the context hidden state corresponding to each token to obtain the emission score. The dimension of the emission score corresponds to the number of label categories and is used to represent the score of each token being labeled as an element label at each address level.

[0054] Further, the emission score, tag ID sequence, and valid token mask are input into the CRF layer. The CRF layer determines the node score for each tag corresponding to each token position based on the emission score, determines the transition score between adjacent tags based on the tag transition score, and limits the range of positions participating in the CRF layer calculation based on the valid token mask. For positions in the valid token mask that correspond to real address characters, the CRF layer includes these positions in the sequence labeling calculation; for positions in the valid token mask that correspond to classification markers, end markers, and padding markers, the CRF layer does not include these positions in the sequence labeling calculation.

[0055] During loss calculation, the CRF layer calculates the sentence-level log-likelihood of the real label path corresponding to the current address text sample based on the emission score, label ID sequence, effective token mask, and label transition score. For a training batch, the sentence-level log-likelihood corresponding to each address text sample in the batch is calculated separately. The negative value of the sentence-level log-likelihood is taken, and the negative log-likelihood of each address text sample in the batch is averaged to obtain the sequence labeling loss.

[0056] During the specific training process, training hyperparameters are set. These hyperparameters include the maximum number of training epochs, batch size, learning rate, weight decay, warm-up ratio, gradient accumulation steps, and maximum norm of gradient clipping. In one specific implementation, the maximum number of training epochs is set to 100, the batch size to 32, the learning rate to 1e-5, the weight decay to 0.01, the warm-up ratio to 0.05, the gradient accumulation steps to 1, and the maximum norm of gradient clipping to 1.0.

[0057] The AdamW optimizer is used to update the model parameters of the address hierarchy feature classification model. Specifically, the model parameters are divided into a weight decay parameter group and a non-weight decay parameter group. Non-bias parameters and non-LayerNorm.weight parameters are assigned to the weight decay parameter group, while bias parameters and LayerNorm.weight parameters are assigned to the non-weight decay parameter group. A linear learning rate scheduler with warm-up is used to adjust the learning rate. During the initial training phase, the learning rate is linearly increased according to the warm-up ratio, and after the warm-up phase, the learning rate is linearly decayed according to the number of training steps.

[0058] In each training epoch, the training data in the dataset object is traversed. For each batch of training data, the sequence labeling loss is calculated, and backpropagation is performed based on the sequence labeling loss to obtain the gradients corresponding to each model parameter in the address hierarchy feature classification model. The timing of parameter updates is determined based on the gradient accumulation steps, and the gradients are clipped according to the maximum norm before parameter updates. Subsequently, the model parameters are updated through the AdamW optimizer, and the current learning rate is updated through a pre-warmed linear learning rate scheduler.

[0059] During training, the loss value for each training batch or training epoch is recorded, and the current loss value is compared with the minimum loss value corresponding to the saved model. When the current loss value is less than the minimum loss value corresponding to the saved model, the model parameters of the current address hierarchy feature classification model are saved as the current optimal model. After training, the address hierarchy feature classification model with the minimum loss during training is used as the post-trained address hierarchy feature classification model.

[0060] Specifically, after obtaining the sequence of address level element labels corresponding to the address text to be parsed, each address character and its corresponding label are read sequentially according to the order of the address characters in the address text to be parsed. The label is formed by a combination of boundary identifiers and address element categories. Boundary identifiers include B, I, and O identifiers, and the address element category is used to indicate the address level type to which the current address character belongs.

[0061] During the reading process, the current address element and the current address element category are set. When the label corresponding to the current address character contains the identifier B, the current address character is determined as the starting character of a new address element, and the current address character is written into the current address element. At the same time, the address element category corresponding to the current address character is recorded as the current address element category.

[0062] Continue reading the next address character. When the label corresponding to the next address character contains the 'I' identifier, and the address element category corresponding to the next address character is consistent with the current address element category, it is determined that the next address character belongs to the internal character of the current address element, and the next address character is appended to the current address element. Following this method, subsequent address characters are read sequentially, and address characters consecutively containing the 'I' identifier and having the same address element category are merged into the current address element.

[0063] When the next tag containing the 'B' identifier is read, it is determined that the previous address element has ended. The merged current address element is stored in the structured address hierarchy element according to the current address element category. At the same time, the address character corresponding to the next tag containing the 'B' identifier is used as the starting character of the new current address element, and the address element category corresponding to the next tag containing the 'B' identifier is used as the new current address element category. The merging process of subsequent address characters continues.

[0064] When a tag containing the O identifier is read, it is determined that the address character corresponding to the O identifier does not belong to the address hierarchy elements to be output, and the address character corresponding to the O identifier is not incorporated into the structured address hierarchy elements. If there is a current address element that has not been written before the O identifier is read, the merging of the current address element is ended first, and the current address element is stored in the structured address hierarchy element according to the current address element category.

[0065] After all address characters are read, if there is still a current address element that has not been written, the current address element is stored in the structured address hierarchy element according to the current address element category. Finally, the structured address hierarchy elements are output according to the arrangement order of each address element in the to-be-parsed address text.

[0066] For example, the to-be-parsed address text is "10 Xitucheng Road, Haidian District, Beijing", and its address hierarchy element tag sequence sequentially includes B-city, I-city, I-city, B-district, I-district, I-district, B-road, I-road, I-road, I-road, B-roadno, I-roadno. When B-city is read, "Bei" is taken as the starting character of the city element; continue to read "jing" and "shi" corresponding to I-city and incorporate them into the city element to obtain "Beijing City". When B-district is read, the merging of the city element is ended, and "Hai" is taken as the starting character of the district and county element; continue to read "dian" and "qu" corresponding to I-district and incorporate them into the district and county element to obtain "Haidian District". In the same way, the road "Xitucheng Road" and the house number "No. 10" are obtained. The finally output structured address hierarchy elements include the city "Beijing City", the district and county "Haidian District", the road "Xitucheng Road" and the house number "No. 10".

[0067] A BERT address hierarchy element classification system based on CRF enhancement, comprising: A labeled data generation module, configured to obtain address text samples, label the address characters in the address text samples according to the BIO tag system, to obtain labeled training data, wherein the BIO tag system includes a boundary identifier for indicating the boundary of an address element and an address element category for indicating the address hierarchy type; A model input construction module, configured to construct model input data according to the labeled training data, wherein the model input data includes an address Token sequence, a tag sequence and a valid Token mask, and the valid Token mask is used to limit the positions of address Tokens participating in the calculation of the CRF layer; The model building module is used to build an address-level feature classification model. The address-level feature classification model includes a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer is used to extract the context hidden state of each token in the address token sequence. The linear classification layer is used to generate the emission score of each token corresponding to each label based on the context hidden state. The CRF layer is used to determine the address-level feature label sequence based on the emission score, label transition score, and effective token mask. The model training module is used to train the address-level feature classification model using the model input data, enabling the CRF layer to calculate the sequence labeling loss based on the emission score, label sequence, and effective token mask. The label decoding module is used to obtain the address text to be parsed. The address text to be parsed is input into the trained address hierarchical feature classification model. The CRF layer decodes the label sequence of the address text to be parsed to obtain the address hierarchical feature label sequence corresponding to the address text to be parsed. The structured output module is used to merge address characters in the address text to be parsed based on the boundary markers and address element categories in the address hierarchy element label sequence, thereby obtaining structured address hierarchy elements.

[0068] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0069] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0070] The present invention has been further described above with reference to specific embodiments. However, it should be understood that the specific description herein should not be construed as limiting the nature and scope of the present invention. Various modifications made to the above embodiments by those skilled in the art after reading this specification are all within the scope of protection of the present invention.

Claims

1. A BERT address hierarchy feature classification method based on CRF enhancement, characterized in that, Includes the following steps: Obtain address text samples, and label the address characters in the address text samples according to the BIO labeling system to obtain labeled training data. The BIO labeling system includes boundary identifiers for representing address element boundaries and address element categories for representing address level types. Model input data is constructed based on the labeled training data, wherein the model input data includes an address token sequence, a label sequence, and a valid token mask, and the valid token mask is used to limit the address token positions participating in the CRF layer calculation; An address-level feature classification model is constructed, comprising a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer is used to extract the context hidden state of each token in the address token sequence. The linear classification layer is used to generate emission scores for each tag corresponding to each token based on the context hidden state. The CRF layer is used to determine the address-level feature tag sequence based on the emission scores, tag transition scores, and the effective token mask. The address-level feature classification model is trained using the model input data, and the CRF layer calculates the sequence labeling loss based on the emission score, the label sequence, and the effective token mask. Obtain the address text to be parsed, input the address text to be parsed into the trained address hierarchy element classification model, and decode the label sequence of the address text to be parsed through the CRF layer to obtain the address hierarchy element label sequence corresponding to the address text to be parsed. Based on the boundary identifiers and address element categories in the address hierarchy element label sequence, the address characters in the address text to be parsed are merged to obtain structured address hierarchy elements.

2. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 1, characterized in that, The address characters in the address text sample are labeled according to the BIO tagging system, including: The address character annotation includes a label formed by combining the boundary identifier and the address element category; The boundary identifiers include B identifiers, I identifiers, and O identifiers. The B identifier is used to indicate the starting position of the address element, the I identifier is used to indicate the internal position of the address element, and the O identifier is used to indicate the position of a non-address element. The address element categories include country, provincial-level administrative region, municipal-level administrative region, district / county, development zone, township / street, community, main road, sub-road, house number, sub-road number, point of interest, sub-point of interest, building number, unit, floor number, room number, business / shop name, auxiliary location keywords, redundant information, and unclassifiable information.

3. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 1, characterized in that, The labeled training data is stored in a character-by-character annotation format. The character-by-character annotation format includes annotation records corresponding to multiple address text samples. Each annotation record includes an address character and a label corresponding to the address character. The label is formed by combining the boundary identifier and the address element category. Annotation records corresponding to different address text samples are separated by blank lines.

4. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 1, characterized in that, The model input data is constructed based on the labeled training data, including: The address text sample is tokenized using a pre-trained word segmenter to obtain the address token sequence. A classification marker [CLS] is added to the beginning of the address token sequence, and an end marker [SEP] is added to the end of the address token sequence to obtain the address token sequence with the added markers. The length of the marked address token sequence is uniformly processed according to a preset maximum length. Specifically, when the length of the marked address token sequence is less than the preset maximum length, padding is used to complete the sequence; when the length of the marked address token sequence is greater than the preset maximum length, it is truncated according to the preset maximum length. The address token sequence with uniform length is converted into an input ID sequence, and an attention mask is constructed based on the valid token positions and padding marker positions in the address token sequence with uniform length. The labels in the labeled training data are converted into a sequence of label IDs, and the model input data is constructed based on the sequence of label IDs and the attention mask.

5. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 4, characterized in that, Converting the labels in the labeled training data into a label ID sequence, and constructing the model input data based on the label ID sequence and the attention mask, includes: Set the labels at the positions corresponding to the classification markers, the end markers, and the fill markers to ignore flags; Convert the tag corresponding to the valid token position into a tag ID to obtain the tag ID sequence; The effective token position and the filling marker position are determined based on the attention mask, and the corresponding positions of the classification marker and the end marker are determined based on the ignore identifier in the tag ID sequence. The valid token mask is constructed based on the valid token position, the fill mark position, the corresponding position of the category mark, and the corresponding position of the end mark. The mask value corresponding to the valid token position is the first mask value, and the mask value corresponding to the fill mark position, the corresponding position of the category mark, and the corresponding position of the end mark is the second mask value.

6. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 4, characterized in that, The address-level feature classification model also includes a Dropout regularization layer, which is set between the BERT pre-trained encoding layer and the linear classification layer; The BERT pre-trained encoding layer uses a Chinese pre-trained BERT model. The BERT pre-trained encoding layer receives the input ID sequence and the attention mask, and outputs the context hiding state corresponding to each token. The Dropout regularization layer performs random deactivation processing on the context hidden state according to a preset random deactivation probability to obtain the processed context hidden state. The linear classification layer maps the processed context hidden state to the emission score of each token corresponding to each tag; The CRF layer determines the address level element tag sequence based on the transmission score and the tag transfer score.

7. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 5, characterized in that, Training the address hierarchy feature classification model using the input data of the model includes: The input ID sequence and the attention mask are input into the BERT pre-trained encoding layer to obtain the context hidden state corresponding to each token. The context-hidden state is input into the linear classification layer to obtain the emission score; Input the emission score, the tag ID sequence, and the valid token mask into the CRF layer; The CRF layer calculates the sentence-level log-likelihood based on the emission score, the tag ID sequence, the valid token mask, and the tag transition score; The sequence labeling loss is obtained by taking the negative value of the log-likelihood at the sentence level and averaging it.

8. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 7, characterized in that, Training the address hierarchy feature classification model using the input data of the model further includes: The preprocessed address data is encapsulated into a dataset object and a data loader object, and the batch data is pruned according to the effective token length using a batch pruning function; Set training hyperparameters, including maximum training epochs, training batch size, learning rate, weight decay value, warm-up ratio, gradient accumulation steps, and maximum norm of gradient clipping; The AdamW optimizer is used to update the model parameters of the address hierarchy feature classification model, wherein the model parameters are divided into a weight decay parameter group and a non-weight decay parameter group. The learning rate is adjusted using a linear learning rate scheduler with preheating. The training data in the dataset object is traversed, the sequence labeling loss corresponding to each batch is calculated, the sequence labeling loss is backpropagated, and the address level feature classification model with the minimum loss during the training process is saved.

9. The BERT address hierarchy feature classification method based on CRF enhancement according to claim 1, characterized in that, Based on the boundary markers and address element categories in the address hierarchy element tag sequence, the address characters in the address text to be parsed are merged to obtain structured address hierarchy elements, including: Read the tags corresponding to each address character in the address hierarchy element tag sequence; When the label corresponding to the current address character contains the identifier B, the current address character is used as the starting character of the current address element, and the address element category corresponding to the current address character is used as the current address element category. When the tag corresponding to the next address character contains the I identifier, and the address element category corresponding to the next address character is consistent with the current address element category, the next address character is merged into the current address element. When the next tag containing the 'B' identifier is read, the merging of the current address element ends, and the address character corresponding to the next tag containing the 'B' identifier is used as the starting character of the next address element. When a tag containing the 'O' identifier is read, the address character corresponding to the 'O' identifier is not incorporated into the structured address hierarchy element. The structured address hierarchy elements are output according to the order in which the address elements are arranged in the address text to be parsed.

10. A BERT address hierarchy feature classification system based on CRF enhancement, characterized in that, include: The labeled data generation module is used to acquire address text samples, label the address characters in the address text samples according to the BIO label system, and obtain labeled training data. The BIO label system includes boundary markers for representing address element boundaries and address element categories for representing address level types. The model input construction module is used to construct model input data based on the labeled training data, wherein the model input data includes an address token sequence, a label sequence, and a valid token mask, and the valid token mask is used to limit the address token positions participating in the CRF layer calculation; The model building module is used to build an address-level feature classification model. The address-level feature classification model includes a BERT pre-trained encoding layer, a linear classification layer, and a CRF layer. The BERT pre-trained encoding layer is used to extract the context hidden state of each token in the address token sequence. The linear classification layer is used to generate the emission score of each tag corresponding to each token based on the context hidden state. The CRF layer is used to determine the address-level feature tag sequence based on the emission score, tag transition score, and the effective token mask. The model training module is used to train the address-level feature classification model using the model input data, so that the CRF layer calculates the sequence labeling loss based on the emission score, the label sequence, and the effective token mask; The label decoding module is used to obtain the address text to be parsed, input the address text to be parsed into the trained address hierarchical element classification model, and perform label sequence decoding on the address text to be parsed through the CRF layer to obtain the address hierarchical element label sequence corresponding to the address text to be parsed. The structured output module is used to merge the address characters in the address text to be parsed according to the boundary identifiers and address element categories in the address hierarchy element label sequence, so as to obtain structured address hierarchy elements.

Images