A named entity recognition method based on mixed-scale sentence representation

Abstract

A named entity recognition method based on mixed-scale sentence representation belongs to the field of natural language processing and machine learning, including: constructing a two-dimensional representation of a sentence based on the ability of a two-dimensional sentence representation to construct a multi-scale representation of a span; based on the scale space theory, converting the original two-dimensional sentence representation into a mixed-scale two-dimensional sentence representation through a text scaling operation; constructing a gated integration module, integrating the mixed-scale two-dimensional sentence representation, calculating the prediction results through multi-layer perception, and completing the recognition and classification of named entities. The present invention generates multiple resized sentence representations by designing a specific text scaling operation, effectively encoding the fine and coarse scale semantic features of the sentence; by combining the gated integration mechanism of the mixed-scale sentence representation to improve the discriminability of the neural network, multiple resized representations can be learned in the two-dimensional sentence representation, and effectively support the recognition of named entities with longer or shorter lengths.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and machine learning, and in particular relates to a named entity recognition method based on mixed-scale sentence representation. Background Art

[0002] With the popularization of computers and the rapid development of the Internet, a large amount of information appears in front of people in the form of electronic documents. In order to cope with the severe challenges brought by the information explosion, professional automated tools are urgently needed to extract truly valuable information from massive data, and information extraction came into being. Named entities refer to proper nouns that represent names of people, places, and organizations in texts. As an important semantic knowledge carrier in texts, named entity recognition plays an important role in information extraction. After it was proposed as a subtask of information extraction, it has been highly valued by the academic community and has been widely studied. Its main purpose is to accurately identify and classify proper nouns that represent named entities in texts, and further provide important semantic support for many natural language processing tasks such as automatic question answering, opinion mining, and semantic analysis. Identifying named entities is widely adopted as a basic task to support downstream NLP tasks, such as co-reference resolution, entity relationship extraction, and knowledge graph construction.

[0003] In rule-based and statistical methods, a series of rules can be defined to capture named entities in sentences. These rules mainly rely on features such as vocabulary, grammatical structure, and part of speech to achieve effective recognition of named entities. In addition, by performing statistical training on a large amount of labeled text data, the features of named entities can be extracted. In the subsequent inference stage, by applying this learned knowledge, the model can effectively identify named entities in new text. In recent studies, a sentence can be converted into a two-dimensional (2D) representation, e.g., table filling (F. Ren, L.Zhang, S. Yin, X. Zhao, S. Liu, B. Li, Y. Liu, A novel global feature-oriented relational triple extraction model based on table filling, in:Proceedings of the 2021 Conference on Empirical Methods in Natural LanguageProcessing, 2021, pp. 2646–2656.) or planarized sentence representation (R. Geng, Y. Chen, R.Huang, Y. Qin, Q. Zheng, Planarized sentence representation for nested namedentity recognition, Information Processing & Management 60 (4) (2023)103352.).

[0004] Two-dimensional sentence representation unfolds a semantic plane. It has the ability to unfold the semantic plane and resolve overlapping semantic structures. In the two-dimensional sentence representation, elements represent entity spans in a sentence. Because adjacent elements on the semantic plane are represented as overlapping phrases in a sentence. They contain the same token and share the same contextual features in a sentence. In the two-dimensional sentence representation, there are also two problems: the contrast fading problem and the multi-scale problem. First, because adjacent elements in the two-dimensional sentence representation share the same contextual features, the semantics of the true span is blurred in appearance. In addition, the semantic information of a true language unit is usually represented by a series of spans of different scales. Therefore, the true span can easily disappear into the background environment. Second, in the two-dimensional sentence representation, spans can represent phrases of different lengths. Because the length of a named entity can be 1, they lead to multi-scale problems, and a span can be represented as a single token or an entire sentence. In this case, short named entities and long named entities may suffer from feature sparsity problems or feature disappearance problems, respectively.

[0005] The contrast fading problem is specifically: suppose there is a sentence "I like to eat apples and bananas". In the two-dimensional sentence representation, the two words "apple" and "banana" may share the same contextual features because they both appear in the context of "I like to eat". However, if you want to capture the difference between "apple" and "banana", this shared contextual feature will blur the distance between the two words in the two-dimensional space, which is the contrast fading problem.

[0006] Specifically, the multi-scale problem is: suppose there is a sentence "I like to eat apples and bananas", "apple" is a single token, and "I like to eat" can be regarded as a span, which contains three tokens. In the two-dimensional sentence representation, it is necessary to process these two different scales of information at the same time, which leads to the multi-scale problem, because a method needs to be found to balance the processing of spans of different lengths. For example, if too much attention is paid to a single token, the information of the long span may disappear; conversely, if too much attention is paid to the long span, the information of the short span may be sparse. Summary of the invention

[0007] In order to solve the contrast fading problem and multi-scale problem existing in the existing method of named entity recognition using two-dimensional sentence representation, the present invention provides a named entity recognition method based on mixed-scale sentence representation. The two-dimensional sentence representation has the ability to construct a multi-scale representation of a span. The present invention makes full use of the ability of the two-dimensional sentence representation to construct a multi-scale representation of a span, adopts a scaling operation, and fully learns the advantages of the semantic dependency relationship between spans with multiple scales.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] A method for named entity recognition based on mixed-scale sentence representation of the present invention comprises the following steps:

[0010] Step 1: Based on the ability of the two-dimensional sentence representation to construct a multi-scale representation of span, construct a two-dimensional representation of the sentence;

[0011] Step 2: Based on the scale space theory, the original two-dimensional sentence representation is converted into a mixed-scale two-dimensional sentence representation through text scaling operation;

[0012] Step 3: Build a gated integration module, integrate mixed-scale two-dimensional sentence representation, calculate the prediction results through multi-layer perception, and complete the recognition and classification of named entities.

[0013] Furthermore, a pre-trained language model is used in combination with a multi-head bi-affine mechanism to convert the serialized original sentences into a two-dimensional sentence representation.

[0014] Furthermore, the fine- and coarse-scale semantic features of a sentence are learned through a text scaling operation, encoding different clues between the true span and the context, thus producing semantic features of different scales.

[0015] .Furthermore, the text scaling operation includes a zoom-in operation and a zoom-out operation; for the zoom-in operation, a nearest neighbor interpolation algorithm is used; for the zoom-out operation, an average pooling operation is used.

[0016] Furthermore, the calculation formula for each output in the amplification operation is:

[0017] , where M represents the original two-dimensional sentence representation (without zooming in or out), i,j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation, and the symbol … Indicates a round-down operation. represents the magnification factor;

[0018] After the two-dimensional sentence representation is magnified, the output is :

[0019]

[0020] in, Represents an element in the two-dimensional sentence representation after the upscaling operation.

[0021] Furthermore, the calculation formula for each output in the reduction operation is:

[0022] , represents the two-dimensional sentence representation after the reduction operation, i, j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation; t represents the number of iterations, which is used as a loop variable here, indicating traversal from 0 to -1; f represents the number of iterations, which is used as a loop variable here to indicate traversal from 0 to -1;

[0023] After the two-dimensional sentence representation is reduced, the output is :

[0024]

[0025] in represents an element in the original two-dimensional sentence representation, represents the reduction factor, and All represent from 0 to Add the elements with -1.

[0026] Furthermore, the gated integration module converts the two-dimensional sentence representations with multiple scales into a unified format with the same dimension size through normalization and integration operations, thereby combining the two-dimensional sentence representations with multiple scales.

[0027] Furthermore, for the two-dimensional sentence representation after the reduction operation , the specific calculation formulas of the normalization operation and the integration operation are as follows:

[0028] ;

[0029] For the two-dimensional sentence representation after the magnification operation , the specific calculation formulas of the normalization operation and the integration operation are as follows:

[0030] ;

[0031] in Represents an element after a reduction operation. Represents an element after the enlargement and restoration operation. represents an element in the two-dimensional sentence representation after the magnification operation, represents the magnification factor, and All represent the range from 0 to The elements of are averaged and pooled. represents an element in the two-dimensional sentence representation after the magnification and restoration operation, Represents the restored sentence representation, that is, adjusted by two reverse operations and The size of the representation;

[0032] The final mixed-scale two-dimensional sentence representation formula is:

[0033]

[0034] (1- )

[0035] Where λ represents the comprehensive weight of features of different scales; W2 and W3 both represent learnable parameters; Sigmod represents the activation function; LeakeReLU represents the ReLU activation function, hCLS represents the beginning representation of the input sentence, and M δ ∈R N×N×dv represents the final mixed-scale 2D sentence representation, which encodes both finer-scale and coarser-scale sentence semantics; represents the two-dimensional sentence representation after the magnification operation is restored. represents a two-dimensional sentence representation that keeps the original size unchanged, Represents the two-dimensional sentence representation after the reduction operation is restored.

[0036] The beneficial effects of the present invention are:

[0037] 1. Contrast fading problem: In two-dimensional sentence representation, since adjacent elements share the same contextual features, the semantics of the true span is blurred in appearance, which causes the true span to easily disappear into the background environment. In order to solve the contrast fading problem, the present invention generates multiple resized sentence representations by designing a specific text scaling operation, effectively encoding the semantic features of the fine and coarse scales of the sentence, and solving the contrast fading problem.

[0038] 2. Targeting the multi-scale problem: In a two-dimensional sentence representation, a span can be represented as a single token or an entire sentence, which leads to a multi-scale problem. To solve the multi-scale problem, the present invention proposes a gated integration mechanism combined with mixed-scale sentence representation to improve the discriminability of the neural network. This mechanism can learn multiple resized representations in a two-dimensional sentence representation, effectively supporting the recognition of named entities with longer or shorter lengths. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is an overall flow chart of a named entity recognition method based on mixed-scale sentence representation of the present invention.

[0040] Figure 2A visual representation of the operation of enlarging the size.

[0041] Figure 3 A visualization of the size reduction operation.

[0042] Figure 4 It is a structural diagram between entity spans. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0044] The present invention provides a method for named entity recognition based on mixed-scale sentence representation, which divides named entity recognition into three steps:

[0045] (1) Construct a two-dimensional representation of the sentence;

[0046] (2) Constructing a hybrid multi-scale representation of a two-dimensional sentence;

[0047] (3) Integrate multi-scale sentence representations and use multi-layer perception to calculate the prediction results.

[0048] like Figure 1 As shown, a method for named entity recognition based on mixed-scale sentence representation of the present invention has the following specific implementation steps:

[0049] Step 1: Based on the ability of the two-dimensional sentence representation to construct a multi-scale representation of span, a two-dimensional representation of the sentence is constructed.

[0050] Based on the existing two-dimensional sentence representation method, a pre-trained language model (PLM) is used in combination with a multi-head dual affine mechanism to convert the original sentence into a two-dimensional sentence representation. The expected result is to convert the serialized original sentence representation into a two-dimensional sentence representation. Compared with the sequence representation of the sentence, the two-dimensional sentence representation has a relatively high performance. The present invention effectively exerts the ability of the two-dimensional sentence representation to expand the semantic plane and solve the overlapping semantic structure, and prevents the chaotic tag representation.

[0051] Specific as Figure 1 As shown in the Encoder Module in the figure, the input sentence is fed into the encoder module, which consists of the pre-trained language model BERT, boundary MLP (start MLP, End MLP), and multi-head biaffine decoder. After the input sentence passes through BERT, the sentence representation H is obtained, and then the sentence representation H is input into the start boundary and the end boundary respectively, and the sentence representation that passes the boundary is input into the multi-head biaffine encoder to obtain a two-dimensional representation of the sentence.

[0052] Step 2: Based on the scale space theory, a mixed-scale sentence representation method is proposed. Two specific text scaling operations are designed. Through the text scaling operation, the original two-dimensional sentence representation can be converted into a two-dimensional sentence representation with multiple scales.

[0053] Based on the two-dimensional sentence representation, a text scaling operation is used to learn the scale representation of the sentence, and semantic features of different scales are generated through the text scaling operation. The text scaling operation aims to learn the fine and coarse scale semantic features of the sentence, which can encode different clues between the real span and the context.

[0054] Since adjacent elements in the two-dimensional sentence representation share the same contextual features, the semantics of the true span is ambiguous in appearance. In addition, the semantic information of a true language unit is usually represented by a series of spans of different scales. Therefore, the true span can easily disappear into the background environment. Spans can represent phrases of different lengths, and since a named entity can be of any length, they lead to multi-scale problems. The mixed-scale two-dimensional sentence representation proposed in the present invention solves the contrast fading problem and the multi-scale problem existing in the two-dimensional sentence representation.

[0055] The specific definition of the text zoom operation is:

[0056] Use M∈ represents a two-dimensional sentence representation; the calculation formula for each output in the enlargement operation is:

[0057] , where M represents the original two-dimensional sentence representation (without zooming in or out), i,j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation, and the symbol … Indicates a round-down operation. represents the magnification factor;

[0058] Therefore, after the two-dimensional sentence representation is magnified, the output is , specifically expressed as:

[0059]

[0060] in, Represents an element in the two-dimensional sentence representation after the upscaling operation.

[0061] Each output in the reduction operation is calculated as:

[0062] , represents the two-dimensional sentence representation after the reduction operation, i, j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation; t represents the number of iterations, which is used as a loop variable here, indicating traversal from 0 to -1; f represents the number of iterations, which is used as a loop variable here to indicate traversal from 0 to -1;

[0063] in, represents an element in the original two-dimensional sentence representation, represents the reduction factor, and All represent from 0 to Add the elements with -1.

[0064] Therefore, after the two-dimensional sentence representation is reduced, the output is , specifically expressed as:

[0065]

[0066] Among them, M in and M out Represents the final output of the text scaling operation implemented on the 2D sentence representation.

[0067] The overall text zoom operation is as follows: Figure 1 As shown in the Scale Module in the figure, after the two-dimensional representation of the input sentence X is obtained through the encoding module, three operations are performed on the two-dimensional sentence representation, namely, enlargement operation, unchanged operation, and reduction operation, to obtain three two-dimensional sentence representations of different sizes. The three different representations enter the semantic learning module (the three identical modules after the two-dimensional sentence representation in the figure) to obtain the learned sentence representation.

[0068] Among them, a mixed-scale sentence representation method is adopted, and for the zoom-in operation, a nearest neighbor interpolation algorithm is used, such as Figure 2 As shown, in the original two-dimensional sentence representation, the two-dimensional sentence representation of the sentence can be regarded as a matrix. The nearest neighbor interpolation algorithm is used for each element in the matrix to interpolate around the original element to obtain an enlarged two-dimensional representation.

[0069] Among them, a mixed-scale sentence representation method is adopted, and an average pooling operation is used for the reduction operation, such as Figure 3 As shown in FIG. 1 , in the two-dimensional sentence matrix, the average pooling operation is used to aggregate the elements around the element using the average pooling method to obtain a reduced two-dimensional sentence representation.

[0070] Step 3: Build a gated integration module to combine two-dimensional sentence representations with multiple scales; finally, use multi-layer perception to calculate the prediction results.

[0071] The present invention specifically designs a gated integration module to integrate mixed-scale features. In the gated integration module, two-dimensional sentence representations with multiple scales are converted into a unified format with the same dimensional size, so that the two-dimensional sentence representations with multiple scales can be combined. The gated integration module further integrates the two-dimensional sentence representations with multiple scales, uses multi-layer perception calculation prediction results, and completes the recognition and classification of named entities.

[0072] Specific as Figure 1 The gated integration module can be divided into two steps: normalization and integration. In the normalization operation step, the two-dimensional sentence representations with multiple scales are converted into a unified format with the same dimension size, so that the two-dimensional sentence representations with multiple scales can be combined. In the integration operation step, the goal focuses on filtering redundant information and selecting important clues from different representations.

[0073] Among them, for the two-dimensional sentence representation after the reduction operation , the specific calculation formulas of the normalization operation and the integration operation are as follows:

[0074]

[0075] For the two-dimensional sentence representation after the magnification operation , the specific calculation formulas of the normalization operation and the integration operation are as follows:

[0076]

[0077] in Represents an element after a reduction operation. Represents an element after the enlargement and restoration operation. represents an element in the two-dimensional sentence representation after the magnification operation, represents the magnification factor, and All represent the range from 0 to The elements of are averaged and pooled. represents an element in the two-dimensional sentence representation after the magnification and restoration operation, Represents the restored sentence representation, that is, adjusted by two reverse operations and Indicates the size.

[0078] The final mixed-scale two-dimensional sentence representation formula is:

[0079]

[0080] (1- )

[0081] Where λ represents the comprehensive weight of features of different scales; W2 and W3 both represent learnable parameters; Sigmod represents the activation function; LeakeReLU represents the ReLU activation function, hCLS represents the beginning representation of the input sentence, and M δ ∈R N×N×dv represents the final mixed-scale 2D sentence representation, which encodes both finer-scale and coarser-scale sentence semantics; represents the two-dimensional sentence representation after the magnification operation is restored. represents a two-dimensional sentence representation that keeps the original size unchanged, Represents the two-dimensional sentence representation after the reduction operation is restored.

[0082] The present invention proposes a named entity recognition method based on mixed-scale sentence representation, which combines the characteristics of two-dimensional sentence representation, which has the ability to unfold semantic planes and resolve overlapping semantic structures, with the characteristics of multi-scale space theory, which can effectively encode the fine and coarse-scale semantic features of sentences. It has achieved excellent results in named entity recognition and has a very good use effect.

[0083] The following uses real data to verify the effectiveness of the named entity recognition method based on mixed-scale sentence representation proposed in the present invention.

[0084] The real data selected by the present invention are from the ACE2005 standard data set, the ACE2004 standard data set and the GENIA data set, as well as the resume data set and the microblog data set. First, the data is preprocessed. The ACE2005 standard data set and the ACE2004 standard data set are collected from news agencies, broadcasting companies and blogs, etc., and they are annotated with 7 entity types. The GENIA data set is collected from the biological field based on three medical subject title terms. It includes 5 entity types and contains 2000 abstracts from PubMed. The resume data set and the microblog data set are both Chinese flat NER data sets. The resume comes from the resume abstracts of 1027 senior executives, and there are 8 entity types in the resume data set. The microblog data set is annotated with 3 entity types. Among them, the ACE and GENIA corpora are annotated with nested named entities. The resume data set and the microblog data set only contain named entities with flat structures.

[0085] In the experiments of the present invention, =3 and =3 is set as the default setting. In the ACE2005 standard dataset and the ACE2004 standard dataset, bert_large_base and Robert_large are used as the default PLM. Because the GENIA dataset is a biological dataset, BioBERT trained on biomedical documents is used as the default PLM as in related work. For the Chinese dataset, BERT-Base-Chinese is used to obtain word vector information. The present invention was trained for 50 epochs with a batch size of 32. The Adam optimizer is used with a learning rate of 2e-5. The number of heads of the multi-head dual affine layer is 5, and the hidden size is 200. The convolution kernel of the two-dimensional CNN layer is 3, and the hidden size is 120. The magnification ratio is 3, and the reduction ratio is 1 / 3.

[0086] First, a two-dimensional representation of the sentence is constructed, and then a mixed-scale sentence representation is obtained through step two of the present invention. Finally, the mixed-scale sentence representation is integrated through step three of the present invention, and the prediction result is calculated through multi-layer perception to verify the effectiveness of the present invention. The evaluation criteria use precision (P), recall rate (R) and F1 value, where F=2*P*R / (P+R).

[0087] The present invention first evaluates the model of the present invention based on three English datasets annotated with nested named entities: ACE2004, ACE005, and GENIA datasets. In order to compare related work, the present invention also divides related work into three categories: transition-based models (Trans), decomposition-based models (Decom), and span-based models (Span). In order to make a fair comparison, in the ACE dataset, the present invention uses the same settings as Yan et al. (Yan H, Sun Y, Li X, et al. Anembarrassingly easy but strong baseline for nested named entity recognition[J]. arXiv preprint arXiv:220804534, 2022) to divide the dataset for training, development, and testing. In the GENIA dataset, the present invention uses the same settings as Li et al. (Li J, Fei H, Liu J, et al. Unified named entity recognition as word-word relation classification; proceedings ofthe proceedings of the AAAI conference on artificial intelligence, F, 2022[C].)

[0088] The same data division. The results are shown in Table 1.

[0089] Table 1

[0090]

[0091] It can be seen from Table 1 that the method for named entity recognition based on mixed-scale sentence representation proposed in the present invention has the following advantages: the present invention organizes all spans of a sentence into a two-dimensional sentence representation, and on this basis proposes a method for learning semantic features at different scales. This method uses different scales of sentence representation to encode, encodes global and local semantic features, has the ability to encode the difference clues between the real span and the background, and effectively improves the ability of the neural network to identify information. In summary, the method for named entity recognition based on mixed-scale sentence representation proposed in the present invention has excellent performance.

[0092] In addition, if Figure 4As shown, (a) represents semantic zoom-in, (b) represents keeping the original two-dimensional sentence representation size unchanged, and (c) represents semantic zoom-out. The bottom layer shows a sentence "Former school leadership team member said" collected from the ACE2004 standard dataset. The sentence is annotated with three overlapping named entities, such as ["school, LOC], ["school leader", PER], ["school leadership team", ORC]. The middle of the bottom layer shows the two-dimensional representation of the sentence, and the two sides are generated by the text zooming and zooming 2D operations of the present invention. The entity representation of the bottom layer also shows the same phenomenon, that is, overlapping named entities with shorter or longer lengths have finer and coarser scale representations in the zoomed-in or zoomed-out scale semantics. Through the text zooming operation, the semantic features of short named entities are enhanced. On the other hand, through the zooming operation, long named entities can have a denser representation to reduce the vanishing gradient problem.

[0093] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for named entity recognition based on mixed-scale sentence representation, characterized in that: The following steps are involved: Step 1: Based on the ability of the two-dimensional sentence representation to construct a multi-scale representation of span, a two-dimensional representation of the sentence is constructed; a pre-trained language model is used in combination with a multi-head dual affine mechanism to convert the serialized original sentence into a two-dimensional sentence representation; Step 2: Based on the scale space theory, the original two-dimensional sentence representation is converted into a mixed-scale two-dimensional sentence representation through a text scaling operation; the text scaling operation includes a zoom-in operation and a zoom-out operation; for the zoom-in operation, a nearest neighbor interpolation algorithm is used; for the zoom-out operation, an average pooling operation is used; The calculation formula for each output in the amplification operation is: Among them, M represents the original two-dimensional sentence representation, i, j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation, and the symbol Indicates the rounding down operation, Z in represents the magnification factor; After the two-dimensional sentence representation is magnified, the output is M in : in, represents an element in the two-dimensional sentence representation after the magnification operation; The calculation formula for each output in the reduction operation is: M out represents the two-dimensional sentence representation after the reduction operation, i, j represents the index of the element subscript in the two-dimensional sentence representation, which is used to locate the element in the two-dimensional sentence representation; t represents the number of iterations, which is used as a loop variable here, indicating traversal from 0 to Z out -1; f represents the number of iterations, which is used as a loop variable here to indicate traversal from 0 to Z out -1; after the two-dimensional sentence is reduced, the output is M out : in represents an element in the original two-dimensional sentence representation, z out represents the reduction factor, and All represent from 0 to Z out Add the elements with -1; Step 3: construct a gated integration module to integrate mixed-scale two-dimensional sentence representations, calculate prediction results through multi-layer perception, and complete the recognition and classification of named entities; the gated integration module converts the two-dimensional sentence representations with multiple scales into a unified format with the same dimensional size through normalization and integration operations, thereby combining the two-dimensional sentence representations with multiple scales.

2. A method for named entity recognition based on mixed-scale sentence representation according to claim 1, characterized in that: Fine- and coarse-scale semantic features of a sentence are learned through a text scaling operation, encoding different clues between the true span and the context, thus producing semantic features at different scales.

3. The method for named entity recognition based on mixed-scale sentence representation according to claim 1, characterized in that: For the two-dimensional sentence representation M after the reduction operation out , the specific calculation formulas of the normalization operation and the integration operation are as follows: For the two-dimensional sentence representation M after the magnification operation in , the specific calculation formulas of the normalization operation and the integration operation are as follows: in Represents an element after a reduction operation. Represents an element after the enlargement and restoration operation. represents an element in the two-dimensional sentence representation after the magnification operation, Z in represents the magnification factor, and All represent the range from 0 to Z in The elements with -1 are averaged pooled. represents an element in the two-dimensional sentence representation after the magnification and restoration operation, represents the restored sentence representation, that is, adjusting M through two reverse operations in and M out The size of the representation; The final mixed-scale two-dimensional sentence representation formula is: λ=Sigmod(W2×LeakeReLU(hCLS×W3)), where λ represents the comprehensive weight of features of different scales; W2 and W3 both represent learnable parameters; Sigmod represents the activation function; LeakeReLU represents the ReLU activation function, hCLS represents the beginning representation of the input sentence, and M δ ∈R N×N×dv represents the final mixed-scale 2D sentence representation, which encodes both finer-scale and coarser-scale sentence semantics; represents the two-dimensional sentence representation after the magnification operation is restored. represents a two-dimensional sentence representation that keeps the original size unchanged, Represents the two-dimensional sentence representation after the reduction operation is restored.

Images