A semantic fusion method and device in a base layer grid governance code

Abstract

The application provides a semantic fusion method and device in basic grid governance coding, and belongs to the technical field of natural language processing. For a first text and a second text to be fused, bidirectional GRU network feature extraction and Transformer-Encoder network coding are respectively performed, the coding results are respectively subjected to average pooling, the pooled results are spliced, the spliced results are subjected to nonlinear conversion, and finally, similarity prediction is performed on the nonlinear converted results to obtain a similarity prediction result of the first text and the second text. If the similarity is high, the two are fused. The application uses a combination of the Transformer-Encoder network and the bidirectional GRU network to match the text, improves the accuracy and efficiency of text matching, and breaks through the limitations of traditional models in context understanding and sequence feature capturing.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a semantic fusion method and apparatus in grassroots grid governance coding. Background Technology

[0002] With the advent of the digital age, urban grassroots governance entities have begun to actively explore digital governance. How to fully leverage the advantages of digital technology and the institutional advantages of urban grassroots governance to empower urban grassroots governance actions and facilitate the digital promotion of the "Fengqiao Experience" nationwide is an important question. In this context, digitally encoding grassroots grid governance elements is the foundation of digital governance and a crucial task. Designing an adaptive, template-based big data dictionary and encoding system for grassroots social grid governance to meet the ever-expanding needs of grassroots management and service operations and digital applications is a major current research topic. One key focus of the encoding system research is the composition of the encoding model. Another important research direction is how to classify coded data information. In the encoding system, how to perform semantic fusion of heterogeneous data is a significant issue. Semantic fusion is the process of unifying the encoding classification of multi-source heterogeneous data from multiple data sources, along with their feature layers, decision layers, and other information. In different data streams, data of the same encoding class may have various different representations. How to quickly identify and perform semantic fusion is one of the key issues in improving encoding efficiency. Semantic fusion is essentially short text matching, and its research goal is to determine whether two text statements are semantically identical by measuring the representation of the statements and their degree of similarity.

[0003] Short text matching is a widely used natural language processing (NLP) technique aimed at modeling the semantic relationships between two texts. As an important branch of NLP, short text matching primarily involves evaluating and modeling the semantic relevance between two text segments. With the rapid development of the internet and social media, massive amounts of text data are generated daily, especially in short text formats such as search queries, social media posts, and short messages. These short texts often lack sufficient contextual information, making them more difficult to understand and process. Short text matching technology aims to address this problem by evaluating and modeling the semantic relevance between two text segments, thereby improving the efficiency and accuracy of information retrieval. Short text matching research faces two main challenges: semantically identical texts with different literal meanings, and texts with different literal meanings but different semantics, stemming from the ambiguity and diversity of natural language. Current research mainly focuses on enhancing the model's ability to represent the semantics of short texts, exploring solutions from three perspectives: designing model structures, leveraging external knowledge, and adding discriminative features. Therefore, this application proposes a Transformer-Encoder-based short text matcher for semantic fusion in grassroots social grid governance coding. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application proposes a semantic fusion method and apparatus for grassroots grid governance coding.

[0005] Firstly, this application proposes a semantic fusion method for coding in grassroots grid governance, including:

[0006] Obtain the first and second texts to be semantically fused, where the first and second texts exist in different heterogeneous data sources;

[0007] The first text and the second text are respectively input into a bidirectional GRU network for feature extraction, resulting in a first sequence and a second sequence;

[0008] The first sequence and the second sequence are respectively input into the Transformer-Encoder network for encoding to obtain the first code and the second code, respectively;

[0009] The first encoding and the second encoding are respectively input into the average pooling layer for average pooling to obtain the first pooling result and the second pooling result respectively;

[0010] The first pooling result and the second pooling result are respectively input into the fusion layer for splicing to obtain the splicing result;

[0011] The splicing result is input into a feedforward network for nonlinear transformation to obtain the transformation result;

[0012] The conversion result is input into the prediction layer for similarity prediction to obtain the similarity prediction result;

[0013] If the similarity prediction result is greater than the preset threshold, it indicates that the first text and the second text are similar. The first text and the second text are then fused, and the fused result is output.

[0014] If the similarity prediction result is less than or equal to a preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text will not be merged.

[0015] The step of inputting the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first feature sequence and a second feature sequence, includes:

[0016] The first text and the second text are respectively input into the BERT word segmenter for word segmentation processing, and the first tag vector and the second tag vector are obtained respectively.

[0017] The first and second marker vectors are converted into indices in the BERT vocabulary to obtain the first positional encoding and the second positional encoding, respectively.

[0018] The first position code and the second position code are respectively input into a bidirectional GRU network for feature extraction, resulting in a first feature sequence and a second feature sequence;

[0019] Add the first position code to the first feature sequence to obtain the first sequence, and add the second position code to the second feature sequence to obtain the second sequence.

[0020] The step of inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding to obtain the first encoding and the second encoding respectively includes:

[0021] By using three different weight matrices, three different linear transformations are performed on the first sequence or the second sequence to obtain three different linear transformation results.

[0022] Based on the results of three different linear transformations, calculate the self-attention of the first sequence or the second sequence to obtain the first self-attention result and the second self-attention result;

[0023] Based on the first self-attention result and the first sequence, perform element-wise addition to obtain the first addition result;

[0024] Based on the second self-attention result and the second sequence, perform element-wise addition to obtain the second addition result;

[0025] The first addition result and the second addition result are normalized respectively to obtain the first normalized result and the second normalized result. The first normalized result is used as the first code and the second normalized result is used as the second code.

[0026] The self-attention of the first sequence or the second sequence is calculated based on the three different linear transformation results, resulting in the first self-attention result and the second self-attention result, as shown in the following formula:

[0027]

[0028] Among them, Z 1 For the first or second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, and d is the value matrix. k Let be the dimension of the key matrix;

[0029] The first addition result is obtained by performing element-wise addition based on the first self-attention result and the first sequence; the second addition result is obtained by performing element-wise addition based on the second self-attention result and the second sequence, and the calculation formula is as follows:

[0030]

[0031] Among them, Z 1 For the first or second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, and d is the value matrix. k Let X be the dimension of the key matrix, X be the first or second sequence, and Z be the dimension of the key matrix. 2 For either the first or second summation result, Self-Attention(O, P, Q) is the first layer of self-attention, i.e., Z. 1 ;

[0032] The first addition result and the second addition result are normalized respectively to obtain a first normalized result and a second normalized result. The first normalized result is used as the first code, and the second normalized result is used as the second code. The calculation formula is as follows:

[0033]

[0034] Where μ is Z 2 The mean along each feature dimension, σ is Z. 2 Along each feature dimension, standard deviation, a is a learnable scaling factor, b is a learnable bias term, and Z... 2 Z represents either the first sum or the second sum. 3 This represents either the first or second normalization result.

[0035] The feedforward network comprises: N fully connected layers, an activation function, N residual connected layers, and N normalization layers, where N is greater than 3; the concatenation result is input into the feedforward network for nonlinear transformation to obtain the transformation result, including:

[0036] Step S6.1: Input the splicing result into the first fully connected layer to obtain the conversion result;

[0037] Step S6.2: Apply an activation function to the transformation result to obtain the transformation result;

[0038] Step S6.3: Input the transformation result and the conversion result into the first residual connection layer to perform residual connection;

[0039] Step S6.4: Input the residual connection result into the first normalization layer for normalization processing;

[0040] Step S6.5: Input the normalization result into the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connected layers and N normalization layers have been traversed, and the final normalized output is obtained. The final normalized output is used as the transformation result.

[0041] The conversion result is then input into the prediction layer for similarity prediction, and the similarity prediction result is obtained. The calculation formula is as follows:

[0042]

[0043] Among them, Z G The global feature is the globally encoded result obtained by inputting the transformation result into the global encoder. Z represents the local features of each layer, which are the local encoding results obtained by inputting the one-dimensional vector of the transformation result into a single-layer local encoder. Z represents the similarity prediction result, β represents the proportion of global features, and ⊙ represents the combination of local features of each layer.

[0044] Secondly, this application proposes a semantic fusion device for grassroots grid governance coding, comprising:

[0045] The text acquisition module is used to acquire the first and second texts to be semantically fused.

[0046] The feature extraction module is used to input the first text and the second text into a bidirectional GRU network for feature extraction, and obtain the first sequence and the second sequence respectively.

[0047] The text encoding module is used to input the first sequence and the second sequence into the Transformer-Encoder network for encoding, respectively, to obtain the first encoding and the second encoding;

[0048] The average pooling module is used to input the first code and the second code into the average pooling layer for average pooling, and obtain the first pooling result and the second pooling result respectively.

[0049] The result fusion module is used to input the first pooling result and the second pooling result into the fusion layer for splicing to obtain the spliced ​​result;

[0050] The nonlinear transformation module is used to input the splicing result into the feedforward network for nonlinear transformation to obtain the transformation result;

[0051] The similarity prediction module is used to input the conversion result into the prediction layer to predict the similarity and obtain the similarity prediction result. If the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar. The first text and the second text are then merged, and the merged result is output. If the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar. The first text and the second text are not merged.

[0052] Beneficial effects:

[0053] This application proposes a semantic fusion method and apparatus for grassroots grid governance coding. It addresses the semantic fusion problem in heterogeneous data sources by using a combination of Transformer-Encoder network and bidirectional GRU network to match text, thereby improving the accuracy and efficiency of text matching and breaking through the limitations of traditional models in context understanding and sequence feature capture. Attached Figure Description

[0054] Figure 1 A flowchart of a semantic fusion method in grassroots grid governance coding according to an embodiment of this application;

[0055] Figure 2 A schematic diagram of the semantic fusion process in grassroots grid governance coding according to an embodiment of this application;

[0056] Figure 3 A schematic diagram of the semantic fusion device in the coding of grassroots grid governance according to an embodiment of this application. Detailed Implementation

[0057] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0058] In grassroots grid governance, inconsistencies in data processing and recording standards across multiple departments and levels lead to data silos and difficulties in information sharing, thereby affecting the efficiency of policy-making and service delivery. This problem is mainly manifested in the possibility that the same information may be recorded differently across different systems, such as differences in address recording formats, and the common occurrence of errors, duplication, or outdated information in the data due to the lack of unified data entry and processing standards.

[0059] To address these challenges, this application proposes a semantic fusion method and apparatus for grassroots grid governance coding, aiming to solve the semantic fusion problem in heterogeneous data sources. The implementation process of this solution is as follows: First, a bidirectional Gated Recurrent Units (BiGRU) network is used to process different short texts to capture their sequence information, ensuring that the temporal sequence of text data is effectively identified. Then, a Transformer Encoder (a crucial part of the Transformer architecture, primarily used for encoding input sequences) is introduced for secondary encoding to deeply mine word-level features in the text, such as keywords and phrases, enhancing the model's ability to understand the deep semantics of the text. Next, average pooling is used to process the encoded features, optimizing the data representation. Then, the features are concatenated to form a comprehensive feature vector for semantic similarity prediction. Finally, these concatenated encoded features are input into a classifier for semantic similarity prediction, achieving effective data fusion.

[0060] The results show that the combination of BiGRU and Transformer-Encoder significantly improves the ability to capture and integrate textual information from different data sources. This efficient semantic fusion method not only handles the semantic differences brought about by heterogeneous data sources, but also facilitates smoother information flow and sharing between different departments. Furthermore, the unified semantic understanding framework effectively reduces misunderstandings and communication costs, improves the efficiency of cross-departmental collaboration, and enhances data quality and resource efficiency in storage and processing by accurately identifying and correcting data errors and duplications. Therefore, this research provides effective technical support for data standardization and information sharing in grassroots social grid governance, contributing to more precise and efficient policy formulation and service delivery.

[0061] Existing technologies all have some problems, such as: 1. CN112749540A: A text matching method, a training method and apparatus for a text matching model. This method obtains text information pairs to be matched, performs word segmentation on the text information pairs to obtain word segmentation information; uses the word segmentation information as input to the text matching model, and determines a weighted text feature vector of the text information pairs based on the word segmentation information. This weight represents the importance of the word segment corresponding to one text information in the other text information pair; and determines the text matching result of the text information pairs based on the weighted text feature vector. 2. CN111782773A: A text matching method and apparatus based on a cascaded pattern. In this method, a first text matching result is obtained by using text matching rules to perform a first text set and a second text set. A second text matching result is obtained by using a text distribution matching algorithm to perform a second text matching on a third text set and a second text set. The third text set is obtained by removing text from the first text set from the first text set. The third text matching result is obtained by performing a third text matching on the fourth and second text sets using a text matching model. The fourth text set is obtained by removing text from the second text matching result from the third text set. Based on the first to third text matching results, a mapping table is established between the texts in the first and second text sets.

[0062] The methods described above all ignore high-level information in the text and fail to capture word-level features. Research on short text matching can better understand the relationships between words and contextual dependencies, allowing for a multi-level understanding of the text and thus more accurate matching and comparison of short texts. This overcomes the limitations of short text matching in context understanding and sequence feature capture.

[0063] Example 1:

[0064] This embodiment proposes a semantic fusion method in grassroots grid governance coding, such as... Figure 1 , Figure 2 As shown, it includes:

[0065] Step S1: Obtain the first text and the second text to be semantically fused, wherein the first text and the second text exist in different heterogeneous data sources;

[0066] To address the semantic fusion problem in heterogeneous data sources, a semantic fusion method based on Transformer-Encoder is designed for semantic similarity prediction, aiming to formalize the short text matching problem. A method combining Transformer-Encoder and BiGRU is proposed to achieve this goal. To clearly define the problem and solution, mathematical notation is used to describe the key components of the algorithm and their interactions.

[0067] Suppose we have two short texts, a first text S1 and a second text S2, denoted as text sequences S1 = {W1(1), W1(2), ..., W1(n)} and S2 = {W2(1), W2(2), ..., W1(n)}, respectively. n (n)}, where W1(i) represents a word in the first text S1 and W2(i) represents a word in the second text S2.

[0068] Step S2: Input the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first sequence and a second sequence, including:

[0069] Step S2.1: Input the first text and the second text into the BERT word segmenter for word segmentation processing to obtain the first tag vector and the second tag vector respectively;

[0070] In this embodiment, the BERT (Bidirectional Encoder Representation from Transformers) word segmenter is used to segment the first text or the second text, and special tags are added to obtain the first tag vector and the second tag vector.

[0071] Step S2.2: Convert the first tag vector and the second tag vector into indices in the BERT vocabulary to obtain the first positional encoding and the second positional encoding, respectively;

[0072] In this embodiment, the first and second marker vectors are converted into indices in the BERT vocabulary to obtain the encoded sentence. The encoded sentence is then padded or truncated to a fixed length, and finally, the first positional code and the second positional code are obtained. The calculation formula is as follows:

[0073]

[0074] Where pos represents the position of a character in the sentence, and i represents the dimension. In the equation... d is the dimension of the character embedding. The positional encoding used in this embodiment involves using sine and cosine functions of different frequencies. These vectors are processed using the BERT model to obtain the BERT word embedding representation of each sentence, that is, to obtain the first positional encoding and the second positional encoding respectively.

[0075] Step S2.3: Input the first position code and the second position code into the bidirectional GRU network for feature extraction to obtain the first feature sequence and the second feature sequence respectively;

[0076] In this embodiment, character embedding is performed on the input characters. First, two short texts, the first text S1 and the second text S2, are input into the bidirectional GRU (BiGRU) network G to capture sequence features and generate sequence feature representations, the first feature sequence G(S1) and the second feature sequence G(S2).

[0077] Step S2.4: Add the first position code to the first feature sequence to obtain the first sequence, and add the second position code to the second feature sequence to obtain the second sequence.

[0078] In this embodiment, the input that is finally fed into the Transformer-Encoder network for encoding is represented by X. The first position code is added to the first feature sequence to obtain the first sequence, and the second position code is added to the second feature sequence to obtain the second sequence.

[0079] Step S3: Input the first sequence and the second sequence into the Transformer-Encoder network for encoding, respectively, to obtain the first encoding and the second encoding, including:

[0080] Step S3.1: Using three different weight matrices, perform three different linear transformations on the first sequence or the second sequence to obtain three different linear transformation results;

[0081] In this embodiment, a Transformer-Encoder network is used for encoding. A Transformer-Encoder typically includes multiple layers of self-attention mechanisms and a feedforward neural network. The self-attention mechanism allows the model to focus on information at different positions in the input sequence, thereby better understanding the contextual relationships and global dependencies in the text.

[0082] The Transformer-Encoder network in this embodiment uses x t The vector representation of the t-th character, x t The first or second sequence is obtained by adding the positional encoding from step S2.4 to the feature sequence. x t Multiply by the three weight matrices respectively:

[0083] W Q W K W V At this time x t Three linear transformations were performed, resulting in three new vectors: q t k t v t The model considers all vectors x in X. t Perform the same operation, and for each x tThe three generated vectors are concatenated to obtain three large matrices q. t k t and v t Based on the above description, the specific calculation method for the matrix can be obtained as follows:

[0084] O = XW O

[0085] P = XW P

[0086] Q = XW Q

[0087] Where O is the result of the first linear transformation, i.e., the query matrix, K is the result of the second linear transformation, i.e., the key matrix, and Q is the result of the third linear transformation, i.e., the value matrix.

[0088] Step S3.2: Based on the three different linear transformation results, calculate the self-attention of the first sequence or the second sequence to obtain the first self-attention result and the second self-attention result. The calculation formula is as follows:

[0089]

[0090] Among them, Z 1 For the first or second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, and d is the value matrix. k Let be the dimension of the key matrix;

[0091] It's understandable: OK T This represents the dot product of the query matrix and the key matrix. This operation is used to calculate the similarity between the query and each key. k This is the scaling factor for the dot product. Scaling the dot product makes the gradient more stable during training. In high-dimensional spaces, the dot product of vectors can be very large, resulting in a very small gradient after the softmax function, which makes it difficult for the network to learn.

[0092] Step S3.3: Perform element-wise addition based on the first self-attention result and the first sequence to obtain the first addition result;

[0093] Step S3.4: Perform element-wise addition based on the second self-attention result and the second sequence to obtain the second addition result;

[0094] In this embodiment, to address the vanishing gradient problem and improve model accuracy, the Transformer-Encoder network introduces an element-wise addition operation between the output of the previous layer and the input of the current layer. Furthermore, to accelerate model convergence, a normalization layer is introduced into the Transformer-Encoder network to normalize the input data.

[0095] The first addition result is obtained by performing element-wise addition based on the first self-attention result and the first sequence; the second addition result is obtained by performing element-wise addition based on the second self-attention result and the second sequence, and the calculation formula is as follows:

[0096]

[0097] Among them, Z 1 For the first or second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, and d is the value matrix. k Let X be the dimension of the key matrix, X be the first or second sequence, and Z be the dimension of the key matrix. 2 For either the first or second summation result, Self-Attention(O, P, Q) is the first layer of self-attention, i.e., Z. 1 ;

[0098] Step S3.5: Normalize the first sum and the second sum to obtain a first normalized result and a second normalized result, respectively. Use the first normalized result as the first code and the second normalized result as the second code. The calculation formula is as follows:

[0099]

[0100] Where μ is Z 2 The mean along each feature dimension, σ is Z. 2 Along each feature dimension, standard deviation, a is a learnable scaling factor, b is a learnable bias term, and Z... 2 Z represents either the first sum or the second sum. 3 This represents either the first or second normalization result.

[0101] Step S4: Input the first encoding and the second encoding into the average pooling layer for average pooling to obtain the first pooling result and the second pooling result respectively;

[0102] In this embodiment, average pooling is applied to these encoded vectors. In the processing of this embodiment, average pooling typically means calculating the average of the feature values ​​across all time steps for each dimension. If Z... 3 Viewed as a matrix, where rows represent time steps and columns represent features, average pooling is performed along the direction of the time steps. Before applying average pooling, and The sizes are N×D and M×D, respectively. Here, N and M represent the number of time steps, and D represents the feature dimension. Average pooling fixes the text representation. and The size of the vector. After applying average pooling, we obtain the vector. and The size of each will be 1×D. These representations are deep encoded vectors of the first text S1 and the second text S2, containing rich sequence and contextual information.

[0103] Step S5: Input the first pooling result and the second pooling result into the fusion layer for splicing to obtain the splicing result;

[0104] In this embodiment, the main purpose of the fusion layer is to combine the vector representations of two short texts processed by Transformer-Encoder and BiGRU. The fusion layer aims to combine the two vectors to generate a new vector that contains information from both original vectors. The method used is vector concatenation. The fusion function F is a simple concatenation operation, and the fused vector representation is as follows: Where [;] denotes vector concatenation. If and Both are d-dimensional vectors, so the concatenated vector Z 5 It is 2D.

[0105] By concatenating the representation vectors of the two texts, the fusion layer captures the combined information from both texts. Since Transformer and BiGRU have been used in the previous steps, these two vectors not only contain the content of the original text, but also rich contextual and sequence features.

[0106] Step S6: Input the splicing result into the feedforward network for nonlinear transformation to obtain the transformation result;

[0107] The feedforward network comprises: N fully connected layers, an activation function, N residual connected layers, and N normalization layers, where N is greater than 3;

[0108] Step S6.1: Input the splicing result into the first fully connected layer to obtain the conversion result;

[0109] Z 6 =Z 5 ×W1+b1

[0110] Among them, Z 6 Z is the output of a typical fully connected layer, i.e., the transformation result. 5 For the splicing result, W1 is the first weight matrix and b1 is the first bias term;

[0111] Step S6.2: Apply an activation function to the transformation result to obtain the transformation result, calculated as follows:

[0112] Z7 =max(0, Z) 5 ×W1+b1)

[0113] Among them, Z 7 For the transformation result, for Z 6 An activation function was used, primarily to increase the nonlinear characteristics of the network. The activation function was the ReLU function, which has the form max(0,x), setting all negative values ​​to 0.

[0114] Step S6.3: Input the transformation result and the conversion result into the first residual connection layer to perform residual connection. The calculation formula is as follows:

[0115] Z 8 =Z 7 +Z 5

[0116] Among them, the transformation result Z 6 and the transformation result Z 7 For being in the same sublayer, Z 8 This is the result of the residual connection.

[0117] Step S6.4: Input the residual connection result into the first normalization layer for normalization processing. The calculation formula is as follows:

[0118] Z 9 =Normalize(Z) 8 )

[0119] Among them, Z 9 To normalize the results, Add & Normalize (residual connections and layer normalization) are two important components in the Transformer model, working together across various layers to improve training efficiency and performance. Each sub-layer contains residual connections, linking the output of the first layer with the output of the next or deeper layers. Residual connections introduce a pass-through connection across layers, directly adding the input to the network's output, allowing the network to learn a residual function. This addresses the gradient explosion and vanishing gradient problems in deep neural networks.

[0120] Step S6.5: Input the normalization result into the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connected layers and N normalization layers have been traversed, and the final normalized output is obtained. The final normalized output is used as the transformation result.

[0121] It can be understood that: the first fully connected layer, the activation function, the first residual connected layer, and the first normalization layer constitute the first sub-layer; the second fully connected layer, the activation function, the second residual connected layer, and the second normalization layer constitute the second sub-layer, and so on, until the Nth fully connected layer, the activation function, the Nth residual connected layer, and the Nth normalization layer constitute the Nth sub-layer, where the activation function is the same, namely the ReLU function. In this embodiment, the concatenation result is sequentially input into the first sub-layer, the second sub-layer, ... the Nth sub-layer to obtain the output result of the Nth sub-layer, which serves as the final transformation result.

[0122] In this embodiment, to improve the data fit, a feedforward network consisting of multiple fully connected layers is used for nonlinear transformation. Each sub-layer has residual connections and normalization operations, and the overall calculation formula is as follows:

[0123]

[0124] Among them, Z 12 The transformed result is then input into the prediction layer as a feature of the first layer. And make subsequent predictions.

[0125] Step S7: Input the conversion result into the prediction layer to perform similarity prediction and obtain the similarity prediction result; if the similarity prediction result is greater than the preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused together to output the fused result; if the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not fused together.

[0126] The conversion result is then input into the prediction layer for similarity prediction, and the similarity prediction result is obtained. The calculation formula is as follows:

[0127]

[0128] Among them, Z G The global feature is the globally encoded result obtained by inputting the transformation result into the global encoder. Z represents the local features of each layer, which are the local encoding results obtained by inputting the one-dimensional vector of the transformation result into a single-layer local encoder. Z represents the similarity prediction result, β represents the proportion of global features, and ⊙ represents the combination of local features of each layer.

[0129] It is understandable that after obtaining the transformation result Z... 12 Then, the transformation result Z 12 As input to the first-level global encoder Obtain the feature output of the first layer use This is used as input to the second-layer global encoder, and the same operation is performed to obtain the feature output of the second layer. Repeat the above process until the feature output of the last layer is obtained. After passing through the h-layer linear global encoder, the feature output Z of the global encoder is obtained. G .

[0130] For the transformation result Z 12 The vector of each dimension is input into a single-layer local encoder layer for feature extraction, obtaining the local encoding result corresponding to the vector of each dimension.

[0131] The above process in this embodiment is represented by pseudocode as follows:

[0132]

[0133]

[0134] This application provides a semantic fusion method for coding grassroots grid governance, specifically a short text matcher based on Transformer-Encoder for semantic fusion in coding grassroots social grid governance. This semantic fusion method uses Transformer-Encoder for secondary encoding, leveraging the Transformer's ability to fuse higher-order text information to capture word-level features. Different short texts are input into a bidirectional GRU network to capture the text's sequence information. Based on BiGRU encoding, Transformer-Encoder is used for secondary encoding, followed by average pooling and concatenation. Finally, the concatenated encoded features are fed into a classifier for final semantic similarity prediction. Compared to traditional bag-of-words models and simple word embedding methods that cannot fully capture the relationships and contextual dependencies between words, the Transformer architecture effectively solves this problem through a self-attention mechanism. This semantic fusion method targets heterogeneous data sources and can understand text from multiple levels, thereby more accurately matching and comparing short texts. This model structure, which combines Transformer and RNN, provides high flexibility for handling various text matching tasks, improves the accuracy and efficiency of short text matching, and solves the limitations of traditional models in context understanding and sequence feature capture.

[0135] The key point of this application is:

[0136] (1) Using the Transformer-Encoder model for short text matching has significant advantages, mainly because this model can effectively address the shortcomings of some traditional methods in handling text matching problems and provides better contextual understanding. The Transformer-Encoder model can capture deep contextual information of each word, which is particularly important for understanding the semantics of short text. Traditional bag-of-words models and simple word embedding methods cannot fully capture the relationships and contextual dependencies between words, while the Transformer architecture effectively solves this problem through a self-attention mechanism.

[0137] (2) The short text matcher based on Transformer-Encoder proposed in this application addresses the semantic fusion problem in heterogeneous data sources and provides a specific scenario design for short text matching, thereby improving the accuracy and efficiency of short text matching.

[0138] (3) Capturing Sequence Features: As a variant of RNN, BiGRU can process sequence data and take into account temporal information in text. Compared with standard RNN, GRU (including BiGRU) solves the long-term dependency problem through gating mechanism, which makes the model more efficient in processing text sequences, especially in short texts, where capturing temporal information is crucial for understanding the overall semantics.

[0139] (4) Enhanced feature extraction capabilities: The combination of Transformer-Encoder and BiGRU utilizes two powerful neural network architectures, effectively combining their respective advantages. This combination provides a very powerful approach to feature extraction, enabling the understanding of text at multiple levels, thus allowing for more accurate matching and comparison of short texts.

[0140] (5) The model addresses the semantic fusion problem in heterogeneous data sources. The combined effect of BERT's multi-head self-attention mechanism and BiGRU's bidirectional processing enables the model to model complex semantic relationships in heterogeneous data sources. Application models are designed for specific scenarios, and using these models can, to a certain extent, advance the development and progress of the short text matching field.

[0141] (6) The method of this application can be integrated into various applications, which improves the flexibility and applicability of the application.

[0142] (7) The method of this application can automatically process and merge text from different data sources, reducing the need for manual annotation and preprocessing, thereby reducing the complexity and related costs of data processing.

[0143] (8) Because it is pre-trained on a wide range of corpora, BERT is able to generalize better to data from different domains.

[0144] Example 2:

[0145] This embodiment proposes a semantic fusion device for coding in grassroots grid governance, such as... Figure 3 As shown, it includes: a text acquisition module, a feature extraction module, a text encoding module, an average pooling module, a result fusion module, a nonlinear transformation module, and a similarity prediction module, which are connected sequentially.

[0146] The text acquisition module is used to acquire the first and second texts to be semantically fused.

[0147] The feature extraction module is used to input the first text and the second text into a bidirectional GRU network for feature extraction, and obtain the first sequence and the second sequence respectively.

[0148] The text encoding module is used to input the first sequence and the second sequence into the Transformer-Encoder network for encoding, respectively, to obtain the first encoding and the second encoding;

[0149] The average pooling module is used to input the first code and the second code into the average pooling layer for average pooling, and obtain the first pooling result and the second pooling result respectively.

[0150] The result fusion module is used to input the first pooling result and the second pooling result into the fusion layer for splicing to obtain the spliced ​​result;

[0151] The nonlinear transformation module is used to input the splicing result into the feedforward network for nonlinear transformation to obtain the transformation result;

[0152] The similarity prediction module is used to input the conversion result into the prediction layer to predict the similarity and obtain the similarity prediction result. If the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar. The first text and the second text are then merged, and the merged result is output. If the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar. The first text and the second text are not merged.

[0153] The various embodiments in the application are described in a progressive manner. For the same or similar parts between the various embodiments, refer to each other. Each embodiment focuses on describing the differences from other embodiments.

[0154] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A semantic fusion method in grassroots grid governance coding, characterized in that, include: Obtain the first and second texts to be semantically fused, where the first and second texts exist in different heterogeneous data sources; The first text and the second text are respectively input into a bidirectional GRU network for feature extraction, resulting in a first sequence and a second sequence; The first sequence and the second sequence are respectively input into the Transformer-Encoder network for encoding to obtain the first code and the second code, respectively; The first encoding and the second encoding are respectively input into the average pooling layer for average pooling to obtain the first pooling result and the second pooling result respectively; The first pooling result and the second pooling result are respectively input into the fusion layer for splicing to obtain the splicing result; The splicing result is input into a feedforward network for nonlinear transformation to obtain the transformation result; The conversion result is input into the prediction layer for similarity prediction to obtain the similarity prediction result; If the similarity prediction result is greater than the preset threshold, it indicates that the first text and the second text are similar. The first text and the second text are then fused, and the fused result is output. If the similarity prediction result is less than or equal to a preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text will not be merged. The feedforward network comprises: N fully connected layers, an activation function, N residual connected layers, and N normalization layers, where N is greater than 3; the concatenation result is input into the feedforward network for nonlinear transformation to obtain the transformation result, including: Step S6.1: Input the splicing result into the first fully connected layer to obtain the conversion result; Step S6.2: Apply an activation function to the transformation result to obtain the transformation result; Step S6.3: Input the transformation result and the conversion result into the first residual connection layer to perform residual connection; Step S6.4: Input the residual connection result into the first normalization layer for normalization processing; Step S6.5: Input the normalization result into the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connected layers and N normalization layers are traversed, and the final normalized output is obtained. The final normalized output is used as the transformation result. The conversion result is then input into the prediction layer for similarity prediction, and the similarity prediction result is obtained. The calculation formula is as follows: ； in, The global feature is the globally encoded result obtained by inputting the transformation result into the global encoder. For each layer, the local features are the one-dimensional vectors of the transformation result input into a single-layer local encoder to obtain the local encoding result. For similarity prediction results, The proportion of global features This indicates that local features of each layer are combined.

2. The semantic fusion method in grassroots grid governance coding according to claim 1, characterized in that, The step of inputting the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first feature sequence and a second feature sequence, includes: The first text and the second text are respectively input into the BERT word segmenter for word segmentation processing, and the first tag vector and the second tag vector are obtained respectively. The first and second marker vectors are converted into indices in the BERT vocabulary to obtain the first positional encoding and the second positional encoding, respectively. The first position code and the second position code are respectively input into a bidirectional GRU network for feature extraction, resulting in a first feature sequence and a second feature sequence; Add the first position code to the first feature sequence to obtain the first sequence, and add the second position code to the second feature sequence to obtain the second sequence.

3. The semantic fusion method in grassroots grid governance coding according to claim 1, characterized in that, The step of inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding to obtain the first encoding and the second encoding respectively includes: By using three different weight matrices, three different linear transformations are performed on the first sequence or the second sequence to obtain three different linear transformation results. Based on the results of three different linear transformations, calculate the self-attention of the first sequence or the second sequence to obtain the first self-attention result and the second self-attention result; Based on the first self-attention result and the first sequence, perform element-wise addition to obtain the first addition result; Based on the second self-attention result and the second sequence, perform element-wise addition to obtain the second addition result; The first addition result and the second addition result are normalized respectively to obtain the first normalized result and the second normalized result. The first normalized result is used as the first code and the second normalized result is used as the second code.

4. The semantic fusion method in grassroots grid governance coding according to claim 3, characterized in that, The self-attention of the first sequence or the second sequence is calculated based on the three different linear transformation results, resulting in the first self-attention result and the second self-attention result, as shown in the following formula: ； in, This is either the result of first self-attention or the result of second self-attention. O For querying the matrix, K The key matrix, Q For value matrices, denoted as the dimension of the key matrix.

5. The semantic fusion method in grassroots grid governance coding according to claim 3, characterized in that, The first addition result is obtained by performing element-wise addition based on the first self-attention result and the first sequence; the second addition result is obtained by performing element-wise addition based on the second self-attention result and the second sequence, and the calculation formula is as follows: ； in, This is either the result of first self-attention or the result of second self-attention. O For querying the matrix, K The key matrix, Q For value matrices, Let X be the dimension of the key matrix, and X be either the first sequence or the second sequence. The result of the first addition or the result of the second addition. This is the first layer of self-attention, i.e. .

6. The semantic fusion method in grassroots grid governance coding according to claim 3, characterized in that, The first addition result and the second addition result are normalized respectively to obtain a first normalized result and a second normalized result. The first normalized result is used as the first code, and the second normalized result is used as the second code. The calculation formula is as follows: ； in, for Along the mean of each feature dimension, for Standard deviation along each feature dimension A learnable scaling factor. For learnable bias terms, The result of the first addition or the result of the second addition. This represents either the first or second normalization result.

7. A semantic fusion device for grassroots grid governance coding, implemented using the semantic fusion method for grassroots grid governance coding as described in any one of claims 1 to 6, characterized in that, include: The text acquisition module is used to acquire the first and second texts to be semantically fused. The feature extraction module is used to input the first text and the second text into a bidirectional GRU network for feature extraction, and obtain the first sequence and the second sequence respectively. The text encoding module is used to input the first sequence and the second sequence into the Transformer-Encoder network for encoding, respectively, to obtain the first encoding and the second encoding; The average pooling module is used to input the first code and the second code into the average pooling layer for average pooling, and obtain the first pooling result and the second pooling result respectively. The result fusion module is used to input the first pooling result and the second pooling result into the fusion layer for splicing to obtain the spliced ​​result; The nonlinear transformation module is used to input the splicing result into the feedforward network for nonlinear transformation to obtain the transformation result; The similarity prediction module is used to input the conversion result into the prediction layer to predict the similarity and obtain the similarity prediction result. If the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar. The first text and the second text are then merged, and the merged result is output. If the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar. The first text and the second text are not merged.

Images