Semantic fusion method and device in base grid governance coding

By combining Transformer-Encoder and bidirectional GRU network methods, the semantic fusion problem of heterogeneous data sources in grassroots grid governance is solved, more accurate text matching and information sharing is achieved, and the efficiency and quality of data processing are improved.

CN118981745AActive Publication Date: 2024-11-19NORTHEASTERN UNIV CHINA
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411119339.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-19
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

In grassroots grid governance, semantic fusion in heterogeneous data sources is difficult to achieve, and traditional models have limitations in context understanding and sequence feature capture, resulting in difficulty in data silos and information sharing.

Method used

Using a method combining Transformer-Encoder and bidirectional GRU network, short text features are extracted and encoded, and a comprehensive feature vector is formed through average pooling and splicing, input to the feedforward network for nonlinear transformation, and finally semantic similarity prediction is carried out to achieve Effective fusion of data.

Benefits of technology

It significantly improves the ability to capture and integrate text information in different data sources, reduces misunderstanding and communication costs, improves cross-departmental collaboration efficiency, and corrects data errors, improves data quality and resource efficiency in storage processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981745A_ABST
    Figure CN118981745A_ABST
Patent Text Reader

Abstract

The invention provides a semantic fusion method and device in base layer grid treatment coding, and belongs to the technical field of natural language processing, bidirectional GRU network feature extraction and Transform-Encoder network coding are carried out on a first text and a second text to be subjected to semantic fusion, average pooling is carried out on coding results, the pooled results are spliced, and a first text and a second text which are about to be subjected to semantic fusion are subjected to two-way GRU network feature extraction and Transform-Encoder network coding; and performing non-linear conversion on a spliced result, finally performing similarity prediction on a non-linear conversion result to obtain a similarity prediction result of the first text and the second text, and if the similarity of the first text and the second text is high, performing fusion. For the semantic fusion problem in the heterogeneous data source, the text is matched by combining the Transform-Encoder network and the bidirectional GRU network, so that the text matching accuracy and efficiency are improved, and the limitation of a traditional model on context understanding and sequence feature capture is broken through.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a semantic fusion method and device in grassroots grid governance coding. Background Art

[0002] With the advent of the digital age, urban grassroots governance entities have begun to actively explore digital governance. How to make full use of the advantages of digital technology and the institutional advantages of urban grassroots governance, empower urban grassroots governance actions, and provide assistance for the digital promotion of the "Fengqiao Experience" across the country is an important issue. In this context, digital coding of grassroots grid governance elements is the basis of digital governance and a vital task. Designing an adaptive template-based grassroots social grid governance big data dictionary and coding system to meet the ever-expanding grassroots management and service business and digital application needs is the current main research topic. One of the research focuses of the coding system is the composition of the coding model. Another important research direction is how to classify the coded data information. In the coding system, how to semantically fuse heterogeneous data is a major issue. Semantic fusion is the process of unifying the coding classification of multi-source heterogeneous data from multiple data sources and their feature layer, decision layer and other information. In different data streams, data of the same coding class may have various different forms of representation. How to quickly identify and perform semantic fusion is one of the key issues to improve coding efficiency. Semantic fusion is essentially short text matching, and its research goal is to determine whether two text sentences are semantically identical by measuring the representation of the sentences and their similarity.

[0003] Short text matching is a widely used natural language processing technology, which aims to model the semantic relationship between two texts. As an important branch in the field of natural language processing (NLP), short text matching mainly involves evaluating and modeling the semantic relevance between two texts. With the rapid development of the Internet and social media, a huge amount of text data is generated every day, especially in the form of short texts, such as search queries, social media posts, short messages, etc. These short texts usually lack sufficient contextual information, making it more difficult to understand and process them. The emergence of short text matching technology aims to solve this problem, by evaluating and modeling the semantic relevance between two texts, improving the efficiency and accuracy of information retrieval. There are two main difficulties in the study of short text matching: different words but the same semantics and the same words but different semantics, which is caused by the ambiguity and diversity of natural language. The current research mainly starts from enhancing the semantic representation ability of the model for short texts, and explores solutions from three perspectives: designing the model structure, using external knowledge, and adding difference features. Therefore, this application proposes a short text matcher based on Transformer-Encoder, which should be used for semantic fusion in grassroots social grid governance coding. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present application proposes a semantic fusion method and device in grassroots grid governance coding.

[0005] In the first aspect, the present application proposes a semantic fusion method in grassroots grid governance coding, including:

[0006] Acquire a first text and a second text to be semantically fused, where the first text and the second text exist in different heterogeneous data sources;

[0007] Inputting the first text and the second text into a bidirectional GRU network for feature extraction to obtain a first sequence and a second sequence respectively;

[0008] Inputting the first sequence and the second sequence into the Transformer-Encoder network for encoding respectively, to obtain a first code and a second code respectively;

[0009] Inputting the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result;

[0010] Inputting the first pooling result and the second pooling result into the fusion layer respectively for splicing to obtain a splicing result;

[0011] Inputting the splicing result into a feedforward network for nonlinear transformation to obtain a transformation result;

[0012] Inputting the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result;

[0013] If the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused and the fused result is output;

[0014] If the similarity prediction result is less than or equal to a preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not merged.

[0015] The first text and the second text are respectively input into a bidirectional GRU network for feature extraction to obtain a first feature sequence and a second feature sequence, respectively, including:

[0016] Inputting the first text and the second text into the BERT word segmenter for word segmentation processing respectively, and obtaining a first token vector and a second token vector respectively;

[0017] Convert the first token vector and the second token vector into indices in the BERT vocabulary to obtain a first position code and a second position code, respectively;

[0018] Inputting the first position code and the second position code into a bidirectional GRU network for feature extraction to obtain a first feature sequence and a second feature sequence respectively;

[0019] The first position code is added to the first feature sequence to obtain the first sequence, and the second position code is added to the second feature sequence to obtain the second sequence.

[0020] The step of inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding to obtain a first code and a second code respectively includes:

[0021] Using three different weight matrices, performing three different linear transformations on the first sequence or the second sequence to obtain three different linear transformation results;

[0022] According to the three different linear transformation results, the self-attention of the first sequence or the second sequence is calculated to obtain the first self-attention result and the second self-attention result;

[0023] Perform an element-by-element addition operation on the first self-attention result and the first sequence to obtain a first addition result;

[0024] Perform an element-by-element addition operation on the second self-attention result and the second sequence to obtain a second addition result;

[0025] The first addition result and the second addition result are respectively normalized to obtain a first normalized result and a second normalized result, and the first normalized result is used as a first code, and the second normalized result is used as a second code.

[0026] According to the three different linear transformation results, the self-attention of the first sequence or the second sequence is calculated to obtain the first self-attention result and the second self-attention result. The calculation formula is as follows:

[0027]

[0028] Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix;

[0029] The first addition result is obtained by performing an element-by-element addition operation on the first self-attention result and the first sequence; the second addition result is obtained by performing an element-by-element addition operation on the second self-attention result and the second sequence, and the calculation formula is as follows:

[0030]

[0031] Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix, X is the first sequence or the second sequence, Z 2 is the first addition result or the second addition result, Self-Attention(O, P, Q) is the first layer of self-attention, that is, Z 1 ;

[0032] The first addition result and the second addition result are respectively normalized to obtain a first normalized result and a second normalized result, the first normalized result is used as the first code, and the second normalized result is used as the second code. The calculation formula is as follows:

[0033]

[0034] Where μ is Z 2 The mean along each feature dimension, σ is Z 2 The standard deviation along each feature dimension, a is a learnable scaling factor, b is a learnable bias term, and Z 2 is the first addition result or the second addition result, Z 3 is the first normalization result or the second normalization result.

[0035] The feedforward network includes: N fully connected layers, an activation function, N residual connection layers and N normalization layers, where N is greater than 3; the concatenation result is input into the feedforward network for nonlinear conversion to obtain a conversion result, including:

[0036] Step S6.1: input the concatenation result into the first fully connected layer to obtain a conversion result;

[0037] Step S6.2: transforming the conversion result using an activation function to obtain a transformation result;

[0038] Step S6.3: input the conversion result and the transformation result into the first residual connection layer to perform residual connection;

[0039] Step S6.4: input the residual connection result to the first normalization layer for normalization processing;

[0040] Step S6.5: Input the normalized result to the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connection layers and N normalized layers are traversed to obtain the final normalized output, and the final normalized output is used as the conversion result.

[0041] The conversion result is input into the prediction layer for similarity prediction to obtain a similarity prediction result, and the calculation formula is as follows:

[0042]

[0043] Among them, Z G is a global feature, wherein the global feature is a global encoding result obtained by inputting the conversion result into a global encoder, is the local feature of each layer, the local feature is the local encoding result obtained by inputting the one-dimensional vector of the conversion result into a single-layer local encoder, Z is the similarity prediction result, β is the proportion of the global feature, and ⊙ represents the combination of the local features of each layer.

[0044] In the second aspect, the present application proposes a semantic fusion device in grassroots grid governance coding, comprising:

[0045] A text acquisition module, used to acquire a first text and a second text to be semantically fused;

[0046] A feature extraction module, used for inputting the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first sequence and a second sequence;

[0047] A text encoding module, used for inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding, to obtain a first code and a second code respectively;

[0048] An average pooling module, used to input the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result;

[0049] A result fusion module, used for inputting the first pooling result and the second pooling result into the fusion layer for splicing respectively to obtain a splicing result;

[0050] A nonlinear conversion module, used for inputting the splicing result into a feedforward network for nonlinear conversion to obtain a conversion result;

[0051] The similarity prediction module is used to input the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result; if the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused to output the fused result; if the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not fused.

[0052] Beneficial effects:

[0053] This application proposes a semantic fusion method and device in grassroots grid governance coding. To address the semantic fusion problem in heterogeneous data sources, a method and device that combines a Transformer-Encoder network and a bidirectional GRU network is used to match text, thereby improving the accuracy and efficiency of text matching and breaking through the limitations of traditional models in context understanding and sequence feature capture. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a semantic fusion method in grassroots grid governance coding according to an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a semantic fusion process in a grassroots grid governance coding according to an embodiment of the present application;

[0056] Figure 3 A principle block diagram of a semantic fusion device in a grassroots grid governance coding in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0058] In grassroots grid governance, the inconsistency of data processing and recording standards across multiple departments and levels has led to data silos and difficulty in information sharing, which has affected the efficiency of policy formulation and service provision. This problem is mainly manifested in the fact that the same information may be recorded differently in different systems, such as the difference in the recording form of the address, and due to the lack of unified data entry and processing standards, there are often errors, duplications or outdated information in the data.

[0059] To address these challenges, this application proposes a semantic fusion method and device in grassroots grid governance coding, which aims to address the semantic fusion problem in heterogeneous data sources. The implementation process of this solution is as follows: First, a bidirectional GRU (BiGRU, Bidirectional Gated Recurrent Units) network is used to process different short texts to capture their sequence information and ensure that the temporal nature of text data is effectively identified. Subsequently, the Transformer-Encoder (Transformer Encoder is an important part of the Transformer architecture, mainly used to encode the input sequence) is introduced for secondary encoding to deeply explore the word granularity features in the text, such as keywords and phrases, and enhance the model's ability to understand the deep semantics of the text. Thereafter, the encoded features are processed by average pooling to optimize the data representation, and then the features are spliced ​​to form a comprehensive feature vector for semantic similarity prediction. Finally, these spliced ​​encoded features are input into the classifier for semantic similarity prediction to achieve effective data fusion.

[0060] The results show that the combination of BiGRU and Transformer-Encoder significantly improves the ability to capture and integrate text information from different data sources. This efficient semantic fusion method not only handles the semantic differences brought by heterogeneous data sources, but also makes the information flow and sharing between different departments smoother. In addition, the unified semantic understanding framework effectively reduces misunderstandings and communication costs, improves the efficiency of cross-departmental collaboration, and improves data quality and resource efficiency of storage and processing by accurately identifying and correcting data errors and duplications. Therefore, this study provides effective technical support for data standardization and information sharing in grassroots social grid governance, which helps to make policy making and service provision more precise and efficient.

[0061] There are some problems in the prior art, such as: 1. CN112749540A text matching method, text matching model training method and device, equipment, the method obtains the text information pair to be matched, performs word segmentation processing on the text information pair, and obtains the word segmentation information; the word segmentation information is used as the input of the text matching model, in which the weighted text feature vector of the text information in the text information pair is determined based on the word segmentation information, and the weight represents the importance of the word segmentation corresponding to one text information in the text information pair in the other text information; the text matching result of the text information pair is determined based on the weighted text feature vector. 2. CN111782773A text matching method and device based on cascade mode, in which the first text matching rule is used to perform the first text matching on the first text set and the second text set to obtain the first text matching result. The text distribution matching algorithm is used to perform the second text matching on the third text set and the second text set to obtain the second text matching result, and the third text set is obtained by removing the text in the first text matching result from the first text set. The text matching model is used to perform a third text matching on the fourth text set and the second text set to obtain a third text matching result, wherein the fourth text set is obtained by removing the text in the second text matching result from the third text set. Based on the first to third text matching results, a mapping relationship table between the texts in the first text set and the second text set is established.

[0062] The above methods all ignore the high-order information of the text and capture the feature information of word granularity. The research on short text matching can better understand the relationship between words and contextual dependencies, understand text from multiple levels, and thus more accurately match and compare short texts. It breaks through the limitations of short text matching in context understanding and sequence feature capture.

[0063] Embodiment 1:

[0064] This embodiment proposes a semantic fusion method in grassroots grid governance coding, such as Figure 1 , Figure 2 As shown, including:

[0065] Step S1: obtaining a first text and a second text to be semantically fused, wherein the first text and the second text exist in different heterogeneous data sources;

[0066] Aiming at the problem of semantic fusion in heterogeneous data sources, a semantic fusion method based on Transformer-Encoder is designed to predict semantic similarity, aiming to formalize the short text matching problem, and a method based on the combination of Transformer-Encoder and BiGRU is proposed to achieve this goal. In order to clearly define the problem and solution, mathematical symbols are used to describe the key components of the algorithm and their interactions.

[0067] Assume that there are two short texts, a first text S1 and a second text S2, which are represented by text sequences S1 = {W1(1), W1(2), ..., W1(n)} and S2 = {W2(1), W2(2), ..., W n (n)}, where W1(i) represents a word in the first text S1, and W2(i) represents a word in the second text S2.

[0068] Step S2: Inputting the first text and the second text into a bidirectional GRU network for feature extraction to obtain a first sequence and a second sequence, respectively, including:

[0069] Step S2.1: inputting the first text and the second text into the BERT word segmenter for word segmentation processing respectively, and obtaining a first token vector and a second token vector respectively;

[0070] In this embodiment, a BERT (Bidirectional Encoder Representation from Transformers) word segmenter is used to segment the first text or the second text, and special tags are added to finally obtain a first tag vector and a second tag vector.

[0071] Step S2.2: Convert the first token vector and the second token vector into indexes in the BERT vocabulary to obtain a first position code and a second position code respectively;

[0072] In this embodiment, the first token vector and the second token vector are respectively converted into indices in the BERT vocabulary to obtain an encoded sentence, and the encoded sentence is padded or truncated to a fixed length to finally obtain a first position code and a second position code, and the calculation formula is as follows:

[0073]

[0074] Among them, pos represents the position of the character in the sentence, i represents the dimension, and in the equation, d is the dimension of character embedding. The position encoding used in this embodiment involves using sine and cosine functions of different frequencies, and processing these vectors using the BERT model to obtain the BERT word embedding representation of each sentence, that is, obtaining the first position encoding and the second position encoding respectively.

[0075] Step S2.3: inputting the first position code and the second position code into the bidirectional GRU network for feature extraction to obtain a first feature sequence and a second feature sequence respectively;

[0076] In this embodiment, character embedding is performed on the input characters. First, two short texts, a first text S1 and a second text S2, are input into a bidirectional GRU (BiGRU) network G to capture sequence features and generate sequence feature representations of a first feature sequence G(S1) and a second feature sequence G(S2).

[0077] Step S2.4: Add the first position code to the first feature sequence to obtain a first sequence, and add the second position code to the second feature sequence to obtain a second sequence.

[0078] In this embodiment, the input finally input to the Transformer-Encoder network for encoding is represented by X. That is, the first position code is added to the first feature sequence to obtain the first sequence, and the second position code is added to the second feature sequence to obtain the second sequence.

[0079] Step S3: Inputting the first sequence and the second sequence into the Transformer-Encoder network for encoding respectively, to obtain a first code and a second code respectively, including:

[0080] Step S3.1: using three different weight matrices to perform three different linear transformations on the first sequence or the second sequence to obtain three different linear transformation results;

[0081] In this embodiment, a Transformer-Encoder network is used for encoding, and the Transformer-Encoder usually includes a multi-layer self-attention mechanism and a feedforward neural network. The self-attention mechanism enables the model to pay attention to information at different positions in the input sequence, thereby better understanding the contextual relationship and global dependency in the text.

[0082] The Transformer-Encoder network in this example uses x t The vector representation of the t-th character, x t is the first sequence or the second sequence obtained by adding the position code in step S2.4 to the feature sequence. t Multiply the three weight matrices separately:

[0083] W Q , W K , W V , at this time x t Three linear transformations are performed and three new vectors are obtained: q t , k t , v t The model is for all vectors x in X t Do the same thing and replace each x tThe three generated vectors are connected to obtain three large matrices q t , k t and v t Based on the above description, the specific calculation method of the matrix can be obtained as follows:

[0084] O=XW O

[0085] P=XW P

[0086] Q=XW Q

[0087] Among them, O is the result of the first linear transformation, that is, the query matrix, K is the result of the second linear transformation, that is, the key matrix, and Q is the result of the third linear transformation, that is, the value matrix.

[0088] Step S3.2: According to the three different linear transformation results, the self-attention of the first sequence or the second sequence is calculated to obtain the first self-attention result and the second self-attention result. The calculation formula is as follows:

[0089]

[0090] Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix;

[0091] It is understandable that: O.K T represents the dot product of the query matrix and the key matrix. This operation is used to calculate the similarity between the query and each key. k is the factor by which the dot product is scaled. Scaling the dot product makes the gradient more stable during training. In high-dimensional space, the dot product of the vectors can be very large, resulting in very small gradients after the softmax function, which makes it difficult for the network to learn.

[0092] Step S3.3: performing an element-by-element addition operation on the first self-attention result and the first sequence to obtain a first addition result;

[0093] Step S3.4: performing an element-by-element addition operation on the second self-attention result and the second sequence to obtain a second addition result;

[0094] In this embodiment, in order to solve the gradient vanishing problem and improve the accuracy of the model, the Transformer-Encoder network introduces an element-by-element addition operation between the output of the previous layer and the input of the current layer. In addition, in order to speed up the convergence of the model, a normalization layer is introduced in the Transformer-Encoder network to normalize the input data.

[0095] The first addition result is obtained by performing an element-by-element addition operation on the first self-attention result and the first sequence; the second addition result is obtained by performing an element-by-element addition operation on the second self-attention result and the second sequence, and the calculation formula is as follows:

[0096]

[0097] Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix, X is the first sequence or the second sequence, Z 2 is the first addition result or the second addition result, Self-Attention(O, P, Q) is the first layer of self-attention, that is, Z 1 ;

[0098] Step S3.5: Normalize the first addition result and the second addition result respectively to obtain a first normalized result and a second normalized result respectively, use the first normalized result as the first code, and use the second normalized result as the second code, and the calculation formula is as follows:

[0099]

[0100] Where μ is Z 2 The mean along each feature dimension, σ is Z 2 The standard deviation along each feature dimension, a is a learnable scaling factor, b is a learnable bias term, and Z 2 is the first addition result or the second addition result, Z 3 is the first normalization result or the second normalization result.

[0101] Step S4: inputting the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result;

[0102] In this embodiment, an average pooling operation is applied to these encoding vectors. In the process of this embodiment, average pooling generally means calculating the average of the feature values ​​of all time steps for each dimension. 3 As a matrix, where the rows represent time steps and the columns represent features, average pooling is performed along the direction of the time step. Before applying average pooling, and The sizes of are N×D and M×D respectively. N and M represent the number of time steps, and D represents the feature dimension. Average pooling fixes the text representation and After applying average pooling, we get the vector and The size of will be 1 × D. These representations are deep encoding vectors of the first text S1 and the second text S2, containing rich sequence and context information.

[0103] Step S5: inputting the first pooling result and the second pooling result into the fusion layer respectively for splicing to obtain a splicing result;

[0104] In this embodiment, in the fusion layer, the main purpose is to combine the vector representations of two short texts processed by Transformer-Encoder and BiGRU. The purpose of the fusion layer is to combine the two vectors to generate a new vector, which should contain the information of the two original vectors. The method used is vector concatenation. The fusion function F is a simple concatenation operation, and the fused vector is represented as: Where [;] means vector concatenation. and are all d-dimensional vectors, then the concatenated vector Z 5 It is 2D dimensional.

[0105] By concatenating the representation vectors of the two texts, the fusion layer captures the comprehensive information of the two texts. Since Transformer and BiGRU have been used in the previous steps, the two vectors contain not only the content of the original text, but also rich context and sequence features.

[0106] Step S6: inputting the splicing result into a feedforward network for nonlinear conversion to obtain a conversion result;

[0107] The feedforward network includes: N fully connected layers, an activation function, N residual connection layers and N normalization layers, where N is greater than 3;

[0108] Step S6.1: input the concatenation result into the first fully connected layer to obtain a conversion result;

[0109] Z 6 =Z 5 ×W1+b1

[0110] Among them, Z 6 is the output of a typical fully connected layer, i.e., the conversion result, Z 5 is the concatenation result, W1 is the first weight matrix, and b1 is the first bias term;

[0111] Step S6.2: The conversion result is transformed using an activation function to obtain a transformation result, which is calculated as follows:

[0112] Z7 =max(0, Z 5 ×W1+b1)

[0113] Among them, Z 7 is the transformation result, for Z 6 The activation function is used to increase the nonlinear characteristics of the network. The activation function is the ReLU function, which is in the form of max(0,x), setting all negative values ​​to 0.

[0114] Step S6.3: Input the conversion result and the transformation result into the first residual connection layer to perform residual connection. The calculation formula is as follows:

[0115] Z 8 =Z 7 +Z 5

[0116] Among them, the conversion result Z 6 And the transformation result Z 7 For the same sublayer, Z 8 is the residual connection result;

[0117] Step S6.4: Input the residual connection result to the first normalization layer for normalization. The calculation formula is as follows:

[0118] Z 9 =Normalize(Z 8 )

[0119] Among them, Z 9 To normalize the results, in the Transformer model, Add&Normalize (residual connection and layer normalization) are two important components that work together in each layer of the model to improve the training efficiency and performance of the model. Each sub-layer has a residual connection, which connects the output of the first layer to the output of the next layer or deeper layer. The residual connection introduces a cross-layer direct connection, which directly adds the input to the output of the network, so that the network learns the residual function, which is used to solve the gradient explosion and gradient disappearance problems in deep neural networks.

[0120] Step S6.5: Input the normalized result to the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connection layers and N normalized layers are traversed to obtain the final normalized output, and the final normalized output is used as the conversion result.

[0121] It can be understood that: the first fully connected layer, the activation function, the first residual connection layer and the first normalization layer are the first sublayer, the second fully connected layer, the activation function, the second residual connection layer and the second normalization layer are the second sublayer, and so on, the Nth fully connected layer, the activation function, the Nth residual connection layer and the Nth normalization layer are the Nth sublayer, where the activation function is the same activation function, that is, the ReLU function. In this embodiment, the splicing result is sequentially input into the first sublayer, the second sublayer...the Nth sublayer to obtain the output result of the Nth sublayer as the final conversion result.

[0122] In this embodiment, in order to improve the fit of the data, a feedforward network consisting of multiple fully connected layers is used for nonlinear transformation. Each sublayer has a residual connection and normalization operation, and the overall calculation formula is as follows:

[0123]

[0124] Among them, Z 12 To convert the result, input it into the prediction layer as the feature of the first layer and make subsequent predictions.

[0125] Step S7: input the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result; if the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused, and the fused result is output; if the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not fused.

[0126] The conversion result is input into the prediction layer for similarity prediction to obtain a similarity prediction result, and the calculation formula is as follows:

[0127]

[0128] Among them, Z G is a global feature, wherein the global feature is a global encoding result obtained by inputting the conversion result into a global encoder, is the local feature of each layer, the local feature is the local encoding result obtained by inputting the one-dimensional vector of the conversion result into a single-layer local encoder, Z is the similarity prediction result, β is the proportion of the global feature, and ⊙ represents the combination of the local features of each layer.

[0129] It is understandable that after obtaining the conversion result Z 12 Afterwards, the conversion result Z 12 As the input of the first layer global encoder Get the feature output of the first layer use As the input of the second layer global encoder, and perform the same operation to obtain the feature output of the second layer Repeat the above process until the feature output of the last layer is obtained After passing through the h-layer linear global encoder, the feature output Z of the global encoder is obtained. G .

[0130] For the conversion result Z 12 The vector of each dimension is input into a single-layer local encoder layer for feature extraction to obtain the local encoding result corresponding to the vector of each dimension.

[0131] The above process of this embodiment is expressed as follows using pseudo code:

[0132]

[0133]

[0134] The present application provides a semantic fusion method in grassroots grid governance coding, specifically a semantic fusion method based on a Transformer-Encoder short text matcher for grassroots social grid governance coding, the semantic fusion method uses Transformer-Encoder for secondary encoding, and uses the high-order information characteristics of Transformer that can fuse text to capture the feature information of word granularity, and inputs different short texts into a bidirectional GRU network to capture the sequence information of the text. On the basis of BiGRU encoding, Transformer-Encoder is used for secondary encoding, and average pooling is performed before splicing. Finally, the spliced ​​encoding features are sent to the classifier, and the final semantic similarity prediction is performed. Compared with the traditional bag of words model and simple word embedding method, the relationship between words and contextual dependencies cannot be fully captured, and the Transformer architecture effectively solves this problem through the self-attention mechanism. This semantic fusion method targets heterogeneous data sources and can understand text from multiple levels, thereby more accurately matching and comparing short texts. This model structure that combines Transformer and RNN provides high flexibility for handling various text matching tasks, improves the accuracy and efficiency of short text matching, and solves the limitations of traditional models in context understanding and sequence feature capture.

[0135] The key points of this application are:

[0136] (1) Using the Transformer-Encoder model to match short texts has significant advantages, mainly because this model can effectively solve the shortcomings of some traditional methods in dealing with text matching problems and can better understand the context. The Transformer-Encoder model can capture the deep context information of each word, which is especially important for understanding the semantics of short texts. The traditional bag-of-words model and simple word embedding methods cannot fully capture the relationship between words and contextual dependencies, while the Transformer architecture effectively solves this problem through the self-attention mechanism.

[0137] (2) The Transformer-Encoder-based short text matcher proposed in this application targets the semantic fusion problem in heterogeneous data sources and carries out specific scenario design for short text matching, thereby improving the accuracy and efficiency of short text matching.

[0138] (3) Capturing sequence features: As a variant of RNN, BiGRU is able to process sequence data and consider the temporal information in the text. Compared with the standard RNN, GRU (including BiGRU) solves the long-term dependency problem through the gating mechanism, which makes the model more efficient in processing text sequences, especially in short texts, where the capture of temporal information is critical to understanding the overall semantics.

[0139] (4) Increased feature extraction capabilities: The combination of Transformer-Encoder and BiGRU uses two powerful neural network architectures, effectively combining their respective strengths. This combination provides a very powerful method for feature extraction, capable of understanding text from multiple levels, thereby more accurately matching and comparing short texts.

[0140] (5) The model aims to solve the problem of semantic fusion in heterogeneous data sources. The multi-head self-attention mechanism of BERT and the bidirectional processing of BiGRU work together to enable the model to model complex semantic relationships in heterogeneous data sources. The application model is designed for specific scenarios, and the use of this model can promote the development and progress of the short text matching field to a certain extent.

[0141] (6) The method of the present application can be integrated into various applications, thereby improving the flexibility and scope of application.

[0142] (7) The method of the present application can automatically process and fuse texts from different data sources, reducing the need for manual annotation and preprocessing, thereby reducing the complexity and associated costs of data processing.

[0143] (8) Due to pre-training on a wide range of corpora, BERT can better generalize to data from different domains.

[0144] Embodiment 2:

[0145] This embodiment proposes a semantic fusion device in grassroots grid governance coding, such as Figure 3 As shown, it includes: a text acquisition module, a feature extraction module, a text encoding module, an average pooling module, a result fusion module, a non-linear conversion module, and a similarity prediction module, wherein the text acquisition module, the feature extraction module, the text encoding module, the average pooling module, the result fusion module, the non-linear conversion module, and the similarity prediction module are connected in sequence;

[0146] A text acquisition module, used to acquire a first text and a second text to be semantically fused;

[0147] A feature extraction module, used for inputting the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first sequence and a second sequence;

[0148] A text encoding module, used for inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding, to obtain a first code and a second code respectively;

[0149] An average pooling module, used to input the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result;

[0150] A result fusion module, used for inputting the first pooling result and the second pooling result into the fusion layer for splicing respectively to obtain a splicing result;

[0151] A nonlinear conversion module, used for inputting the splicing result into a feedforward network for nonlinear conversion to obtain a conversion result;

[0152] The similarity prediction module is used to input the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result; if the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused to output the fused result; if the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not fused.

[0153] The various embodiments in the application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments.

[0154] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A semantic fusion method in grassroots grid governance coding, characterized in that: include: Acquire a first text and a second text to be semantically fused, where the first text and the second text exist in different heterogeneous data sources; Inputting the first text and the second text into a bidirectional GRU network for feature extraction to obtain a first sequence and a second sequence respectively; Inputting the first sequence and the second sequence into the Transformer-Encoder network for encoding respectively, to obtain a first code and a second code respectively; Inputting the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result; Inputting the first pooling result and the second pooling result into the fusion layer respectively for splicing to obtain a splicing result; Inputting the splicing result into a feedforward network for nonlinear transformation to obtain a transformation result; Inputting the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result; If the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused and the fused result is output; If the similarity prediction result is less than or equal to a preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not merged.

2. The semantic fusion method in grassroots grid governance coding according to claim 1 is characterized in that: The first text and the second text are respectively input into a bidirectional GRU network for feature extraction to obtain a first feature sequence and a second feature sequence, respectively, including: Inputting the first text and the second text into the BERT word segmenter for word segmentation processing respectively, and obtaining a first token vector and a second token vector respectively; Convert the first token vector and the second token vector into indices in the BERT vocabulary to obtain a first position code and a second position code, respectively; Inputting the first position code and the second position code into a bidirectional GRU network for feature extraction to obtain a first feature sequence and a second feature sequence respectively; The first position code is added to the first feature sequence to obtain the first sequence, and the second position code is added to the second feature sequence to obtain the second sequence.

3. The semantic fusion method in grassroots grid governance coding according to claim 1 is characterized in that: The step of inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding to obtain a first code and a second code respectively includes: Using three different weight matrices, performing three different linear transformations on the first sequence or the second sequence to obtain three different linear transformation results; According to the three different linear transformation results, the self-attention of the first sequence or the second sequence is calculated to obtain the first self-attention result and the second self-attention result; Perform an element-by-element addition operation on the first self-attention result and the first sequence to obtain a first addition result; Perform an element-by-element addition operation on the second self-attention result and the second sequence to obtain a second addition result; The first addition result and the second addition result are respectively normalized to obtain a first normalized result and a second normalized result, and the first normalized result is used as a first code, and the second normalized result is used as a second code.

4. The semantic fusion method in grassroots grid governance coding according to claim 3 is characterized in that: According to the three different linear transformation results, the self-attention of the first sequence or the second sequence is calculated to obtain the first self-attention result and the second self-attention result. The calculation formula is as follows: Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix.

5. The semantic fusion method in grassroots grid governance coding according to claim 3 is characterized in that: The first addition result is obtained by performing an element-by-element addition operation on the first self-attention result and the first sequence; the second addition result is obtained by performing an element-by-element addition operation on the second self-attention result and the second sequence, and the calculation formula is as follows: Among them, Z 1 is the first self-attention result or the second self-attention result, O is the query matrix, K is the key matrix, Q is the value matrix, d k is the dimension of the key matrix, X is the first sequence or the second sequence, Z 2 is the first addition result or the second addition result, Self-Attention(O, P, Q) is the first layer of self-attention, that is, Z 1 .

6. The semantic fusion method in grassroots grid governance coding according to claim 3 is characterized in that: The first addition result and the second addition result are respectively normalized to obtain a first normalized result and a second normalized result, the first normalized result is used as the first code, and the second normalized result is used as the second code. The calculation formula is as follows: Where μ is Z 2 The mean along each feature dimension, σ is Z 2 The standard deviation along each feature dimension, a is a learnable scaling factor, b is a learnable bias term, and Z 2 is the first addition result or the second addition result, Z 3 is the first normalization result or the second normalization result.

7. The semantic fusion method in grassroots grid governance coding according to claim 1 is characterized in that: The feedforward network includes: N fully connected layers, an activation function, N residual connection layers and N normalization layers, where N is greater than 3; the concatenation result is input into the feedforward network for nonlinear conversion to obtain a conversion result, including: Step S6.1: input the concatenation result into the first fully connected layer to obtain a conversion result; Step S6.2: transforming the conversion result using an activation function to obtain a transformation result; Step S6.3: input the conversion result and the transformation result into the first residual connection layer to perform residual connection; Step S6.4: input the residual connection result to the first normalization layer for normalization processing; Step S6.5: Input the normalized result to the second fully connected layer, and repeat steps S6.1 to S6.4 again until all N fully connected layers, N residual connection layers and N normalized layers are traversed to obtain the final normalized output, and the final normalized output is used as the conversion result.

8. The semantic fusion method in grassroots grid governance coding according to claim 1 is characterized in that: The conversion result is input into the prediction layer for similarity prediction to obtain a similarity prediction result, and the calculation formula is as follows: Among them, Z G is a global feature, wherein the global feature is a global encoding result obtained by inputting the conversion result into a global encoder, is the local feature of each layer, the local feature is the local encoding result obtained by inputting the one-dimensional vector of the conversion result into a single-layer local encoder, Z is the similarity prediction result, β is the proportion of the global feature, and ⊙ represents the combination of the local features of each layer.

9. A semantic fusion device in grassroots grid governance coding, characterized in that: include: A text acquisition module, used to acquire a first text and a second text to be semantically fused; A feature extraction module, used for inputting the first text and the second text into a bidirectional GRU network for feature extraction, respectively, to obtain a first sequence and a second sequence; A text encoding module, used for inputting the first sequence and the second sequence into a Transformer-Encoder network for encoding, to obtain a first code and a second code respectively; An average pooling module, used to input the first code and the second code into the average pooling layer for average pooling, respectively, to obtain a first pooling result and a second pooling result; A result fusion module, used for inputting the first pooling result and the second pooling result into the fusion layer for splicing respectively to obtain a splicing result; A nonlinear conversion module, used for inputting the splicing result into a feedforward network for nonlinear conversion to obtain a conversion result; The similarity prediction module is used to input the conversion result into the prediction layer for similarity prediction to obtain a similarity prediction result; if the similarity prediction result is greater than a preset threshold, it indicates that the first text and the second text are similar, and the first text and the second text are fused to output the fused result; if the similarity prediction result is less than or equal to the preset threshold, it indicates that the first text and the second text are not similar, and the first text and the second text are not fused.

Citation Information

Patent Citations

  • Text similarity determination method and system, storage medium and electronic equipment

    CN112926339A

  • Short text similarity prediction method based on keywords

    CN114528831A

  • Text semantic matching method and device for intelligent questions and answers of fire safety knowledge

    CN114547256A

  • Multi-intention spoken language understanding method based on fluctuation threshold and segmentation

    CN117151121A

  • Information processing method and system based on big data

    CN117236323A