College entrance examination major attribute entity extraction method based on difference attention mechanism

Abstract

The application belongs to the technical field of computer application, and particularly relates to a college major attribute entity extraction method based on a difference attention mechanism. The method takes a character vector, an external vocabulary vector and an entity text vector as keys, values and queries of the difference attention mechanism, effectively integrates the difference of external knowledge into the original text, improves the personalized representation of the character vector in different contexts, realizes better fusion, simultaneously, utilizes a CRF model to perform structure output, and further realizes the college major attribute entity extraction, attributes are assigned to the college majors, so that the candidates can conveniently select.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer application, and particularly relates to a college major attribute entity extraction method based on a difference attention mechanism. BACKGROUND

[0002] At present, the traditional enrollment form of college entrance examination is basically that the relevant information is queried and listed manually, and the school majors meeting the hard conditions are screened. At present, the existing college entrance examination information site provides a search query level enrollment service, and can query information such as school, major enrollment line and ranking position, and can also evaluate and analyze the enrollment colleges and judge the enrollment probability. The ideal colleges for the examinees are provided.

[0003] However, when the examinees fill in the college entrance examination, they have not contacted the society and employment, and the understanding of the majors in the university is not very sufficient, and the self-learning and employment cognition is not clear, so that the major filled in the college entrance examination may not be the real will of the examinees, and the interest and hobby of the examinees cannot be developed. On the other hand, the students cannot measure the focus of the basic skills and knowledge required by the major, and also have a negative impact on the subsequent learning.

[0004] The official text introduction of the university major often cannot make the examinees well understand the specific learning content and employment direction of the major, and therefore, how to accurately give the major attributes to facilitate the examinees to understand the major. SUMMARY

[0005] The application proposes a college major attribute entity extraction method based on a difference attention mechanism, which is simple in method, convenient in operation and can better give the college major attributes by the difference attention mechanism.

[0006] In order to achieve the above purpose, the technical scheme adopted by the application is that the application provides a college major attribute entity extraction method based on a difference attention mechanism, comprising the following steps:

[0007] a. First, collect the university major related content text;

[0008] b. The Chinese pre-training model is used to embed and represent the text to be extracted, and the original text embedding is obtained;

[0009] c. The semantic information of the character vector is enhanced by the way of external knowledge supplement, wherein the external knowledge includes external vocabulary knowledge and external short text semantic knowledge and completes the external knowledge embedding representation;

[0010] d. Then, the external knowledge information is processed by the attention mechanism, integrated into the original text embedding, and then the integrated information is input into the encoder part of the Transformer model for calculation.

[0011] e. The encoder calculation result of the Transformer model is input into the CRF model to complete sequence labeling, that is, the entity extraction of the college major attribute is completed.

[0012] In the d step, the attention scoring function in the processing of the external knowledge information by the attention mechanism is:

[0013]

[0014] Among them, the external short text semantic knowledge embedding is represented as vs, the external vocabulary knowledge embedding is represented as vw, and cat() represents the calculation of the Pytorch.cat() function. Therefore, the final difference attention mechanism calculation is:

[0015]

[0016] Among them, vx represents the original text embedding, and vt represents the result of integrating the external knowledge information into the original text embedding.

[0017] As a preferred, in the d step, the calculation formula to the encoder of the Transformer model is:

[0018] vt`=Transformer encoder (vt)

[0019] Among them, Transformer encoder represents the calculation of the encoder part of the Transformer model.

[0020] As a preferred, in the e step, the result of sequence labeling is:

[0021] result=CRF(vt`)。

[0022] Compared with the prior art, the advantages and positive effects of the present application are that,

[0023] The method takes the character vector, external vocabulary vector and entity text vector as the key, value and query of the difference attention mechanism, effectively integrates the difference of external knowledge into the original text, improves the personalized representation of the character vector in different contexts, realizes better fusion, and at the same time, uses the CRF model for structure output, thereby realizing the entity extraction of the college major attribute, assigning attributes to the college major, and facilitating the selection of the examinee. DETAILED DESCRIPTION

[0024] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below with reference to embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0025] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0026] This embodiment provides a method for extracting entities related to college entrance examination major attributes based on a differential attention mechanism.

[0027] To achieve entity extraction of college entrance examination major attributes, this method mainly includes three sub-modules: text embedding module, knowledge supplementation module, and sequence labeling module. The specific steps are as follows:

[0028] The first step is to compile and collect relevant texts about university majors, including introductions to university majors, job descriptions, and job postings.

[0029] The text embedding module is mainly used to embed text related to university majors.

[0030] Given a sentence of length n, x = [x1, x2, ..., xn], and assign named entity tags to sentence x as y = {y1, y2, ..., yn}, where n is the sequence length, and yi takes values ​​of B_entityType, I_entityType, and O mode. entityType represents the entity type to be extracted, including major name, discipline name, job title name, core course name, and professional knowledge name. In this embodiment, the Roberta Chinese pre-trained model is used for embedding representation of the text to be extracted. The calculation process is as follows:

[0031] vx = Roberta(x)

[0032] Roberta(x) represents taking the output vector value of the last layer of the Roberta Chinese pre-trained model.

[0033] The knowledge supplementation module primarily embeds external information knowledge. This is mainly to address the limited sample data and enhance the semantic information of character vectors through external knowledge supplementation. This method uses two types of external knowledge: external lexical knowledge (W = [w1, w2, ..., w...)). j ,....w m ]) and external short text semantic knowledge (S=[s1,s2,...,si ,....s n ]) where m and n represent the number of lexical and semantic short texts respectively, and the external knowledge is mainly obtained through entity text explanation, external dictionary, etc.

[0034] For example, there is input text x = ['count', 'computer', 'industry','main', 'from', 'things', 'industry', 'what', '?'], y = [B_spe, I_spe, I_spe, O, O, O, O, O, O, O, O, O, O, O, O], at this time, the external vocabulary, i.e. the external short text, is matched, and W x = ['computer', 'computer application technology',...], S x = [s1, s2,...]. Where:

[0035] s1 = 'Computer is commonly known as computer, a modern electronic computer machine for high-speed calculation, which can perform numerical calculation and logic calculation, and also has storage memory function. It is a modern intelligent electronic device that can run according to program, automatically and high-speed process mass data.'

[0036] s2 = 'Computer application technology is a general higher vocational professional, belonging to computer class, with a basic study period of three years.'

[0037] The external lexical knowledge part is embedded through the pre-trained word vector based on word2vec, with

[0038] vw = Word2vec(w j )

[0039] The external short text semantic knowledge level is embedded through the Roberta Chinese pre-training model, with

[0040] vs = RoBerta(s i )

[0041] Sequence labeling module

[0042] The above two modules complete the embedding representation of the basic data, and this module fully learns the embedding representation of the data to realize the entity extraction task. In addition to the text embedding module and the knowledge supplement module, this part mainly uses two kinds of deep learning models, which are attention mechanism model and CRF model. Through the attention mechanism, the external information of the knowledge supplement module is processed, and the original text embedding vx of the data is fused, and the fused result is recorded as vt, with

[0043] vt = Att((K, V), Q)

[0044] Att() represents the difference attention mechanism. First, define the attention mechanism calculation process as:

[0045]

[0046] wherein,

[0047] α n =softmax(s(K,Q))

[0048] represents the attention distribution, and at the same time:

[0049]

[0050] is the attention scoring function, which is the calculation method of self-attention mechanism. In this method, the input text character vector representation vx is defined as V, the external knowledge word vector representation vw is equivalent to K, the entity short text explanation vector in the input text vs is equivalent to Q. For the embedding dimension of each part (Q, K, V), it is set to 256. For the calculation of attention, the entity in the data that matches the external word can directly obtain Q, K, V to participate in the calculation, and for the entity that cannot obtain the external word or the external short text semantic, the self-attention mechanism will be calculated to ensure the symmetry of the calculation process.

[0051] The attention mechanism calculation method involved in this method is different from the conventional attention mechanism. In this method, the attention scoring function s is optimized, and its calculation process is optimized as:

[0052]

[0053] wherein cat() represents the calculation of Pytorch.cat() function, that is, the concatenation of two vectors. This improvement is because the entity vector and the external word vector are obtained in different training ways to have local obvious vectors, and also fuse semantic information, so the fused vector is used to calculate the attention with the short text semantic vector of the entity, which can well let the character obtain personalized vector representation. The difference attention mechanism calculation is:

[0054]

[0055] The above information is input to the downstream model for calculation. There are

[0056] vt=AttD((vw,vs),vx)

[0057] vt`=Transformer encoder (vt)

[0058] wherein wherein, Transformerencoder The encoder part of the Transformer model is represented to perform calculation. Finally, vt` is input into the CRF to complete sequence labeling.

[0059] result = CRF(vt`)

[0060] The final output result is the labeling result of the input text sequence, that is, the extraction of the professional attribute entity of the college entrance examination can be completed.

[0061] The above is only a preferred embodiment of the present application, not a limitation on other forms of the present application. Any skilled person in the art can use the disclosed technical content to make changes or modifications to equivalent embodiments applied to other fields. However, any simple modification, equivalent change and modification made on the basis of the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still belongs to the protection scope of the technical solution of the present application.

Claims

1. A method for extracting entities related to college entrance examination major attributes based on differential attention mechanism, characterized in that, Includes the following steps: a. First, collect texts related to university majors; b. Use a Chinese pre-trained model to embed the extracted text to obtain the original text embedding; c. Enhance the semantic information of character vectors by supplementing them with external knowledge, where external knowledge includes external lexical knowledge and external short text semantic knowledge, and complete the external knowledge embedding representation; d. Then, external knowledge information is processed through an attention mechanism and integrated into the original text embedding. After the information is integrated, it is input into the encoder part of the Transformer model for computation. e. Input the encoder calculation results of the Transformer model into the CRF model to complete the sequence labeling, that is, complete the entity extraction of the college entrance examination major attributes. In step d, the attention scoring function for processing external knowledge information through the attention mechanism is: ， Among them, the semantic knowledge embedding of external short texts is represented as External lexical knowledge embedding representation is , This represents the computation of the PyTorch.cat() function. Therefore, the final differential attention mechanism computation is as follows: ， in, Indicates the embedding of the original text. This represents the result of embedding external knowledge information into the original text, where the input text character vector represents... Let V be the external knowledge vocabulary vector representation. Equivalent to K, the short text interpretation vector of entities in the input text. Equivalent to Q.

2. The method for extracting college entrance examination major attribute entities based on differential attention mechanism according to claim 1, characterized in that, In step d, the formula for calculating the encoder to the Transformer model is: ， in, This indicates that the encoder part of the Transformer model is computed.

3. The method for extracting college entrance examination major attribute entities based on differential attention mechanism according to claim 2, characterized in that, In step e, the result of sequence labeling is: 。

Images