Information operation and maintenance professional knowledge retrieval method based on explicit and implicit redundancy elimination representation learning
By constructing an information operation and maintenance expertise retrieval model with explicit and implicit de-redundant representation learning, the problem of redundant information in information operation and maintenance expertise retrieval is solved, and a more efficient information operation and maintenance expertise retrieval effect is achieved.
Patent Information
- Application Number
- CN202410284113.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2025-09-19
AI Technical Summary
Existing information operation and maintenance expertise retrieval methods rely on the representations learned in the model pre-training phase and cannot effectively remove redundant information in information operation and maintenance expertise, resulting in poor retrieval results.
An information operation and maintenance professional knowledge retrieval model was constructed based on explicit and implicit de-redundant representation learning. The model included an underlying semantic representation model, an explicit de-redundant representation learning module, and an implicit de-redundant representation learning module. The model was initialized using the pre-trained language model BERT, and explicit de-redundancy was performed through a one-dimensional dilated convolutional network and a fully connected network. The covariance matrix was used to reduce implicit redundancy, and a triple contrast loss function was constructed for training.
It significantly improves the accuracy and efficiency of information operation and maintenance expertise retrieval, outperforming existing methods, and can better understand the semantic relevance of information operation and maintenance expertise and reduce the impact of redundant information.
Smart Images
Figure CN120670566A_ABST
Abstract
Description
Technical Field
[0001] The present invention and an information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundancy representation learning belong to the field of electric power technology. Background Art
[0002] Information operations personnel often encounter complex and difficult problems. To improve problem-solving efficiency, we consider constructing an information operations expertise retrieval task. This task aims to retrieve relevant information operations expertise from large open-source information operations expertise repositories using natural language queries based on specific requirements.
[0003] Currently, methods for searching for information operation expertise can be broadly categorized as those based on information retrieval (IR), deep learning, and pre-trained language models. Early retrieval methods primarily focused on the relevant parts of knowledge and natural language queries, using IR-based methods to find relevant knowledge. However, these early methods typically performed simple fragment matching between information operation expertise and natural language queries, failing to effectively assess the semantic relevance between the two. With the development of deep learning technology, deep learning-based knowledge retrieval methods have addressed this issue to some extent. These methods typically embed knowledge and natural language queries into a shared high-dimensional vector space and use neural networks to compare their semantic similarity. Recently, a series of pre-trained language models, led by BERT, have gained widespread application. Pre-training on large-scale datasets further improves the models' ability to understand knowledge semantics and also demonstrates excellent transferability to various downstream tasks.
[0004] Although current knowledge retrieval methods based on deep learning and pre-trained language models have achieved certain results, these methods rely almost entirely on the representations learned in the model pre-training stage and return knowledge retrieval results based on the cosine similarity between the expertise representation and the natural language query representation. However, information operation and maintenance expertise is often longer than natural language queries, and there is some redundant information in the information operation and maintenance expertise representation, which also leads to poor performance on some information operation and maintenance expertise retrieval samples. Summary of the Invention
[0005] In order to overcome the problems in the prior art, the present invention proposes an information operation and maintenance professional knowledge retrieval method that can remove information redundancy.
[0006] In order to solve the above technical problems, the present invention proposes a technical solution: a method for retrieving information operation and maintenance expertise based on explicit and implicit de-redundant representation learning, comprising the following steps:
[0007] Step 1: Construct an information operation and maintenance professional knowledge retrieval dataset; collect information operation and maintenance professional knowledge snippets and the natural language questions and answers corresponding to the information operation and maintenance professional knowledge snippets from the information operation and maintenance professional knowledge database; use the information operation and maintenance professional knowledge snippets as information operation and maintenance professional knowledge data and the natural language questions and answers as natural language query data to construct the data pairs of <natural language query data, information operation and maintenance professional knowledge data>, which constitute the information operation and maintenance professional knowledge retrieval dataset;
[0008] Step 2: Construct an information operation and maintenance expertise retrieval model. The information operation and maintenance expertise retrieval model consists of an underlying semantic representation model, an explicit de-redundant representation learning module, and an implicit de-redundant representation learning module. These modules are constructed in sequence.
[0009] Step 3: Train the information operation and maintenance expertise retrieval model. Based on the information operation and maintenance expertise retrieval dataset obtained in step 1 and the information operation and maintenance expertise retrieval model obtained in step 2, set training parameters, construct a loss function, and train the information operation and maintenance expertise retrieval model.
[0010] Step 4: Information operation and maintenance professional knowledge retrieval; using the information operation and maintenance professional knowledge retrieval model trained in step 3, calculate the similarity between the information operation and maintenance questions given by the user and the information operation and maintenance professional knowledge data in the knowledge base; select the group with the highest similarity and push it to the user.
[0011] A further improvement of the above scheme is that: in step 1, the operation and maintenance professional knowledge retrieval data set is divided into several subsets, namely training set, verification set, and test set, and the corresponding information operation and maintenance professional knowledge base files are sorted for use in the verification and testing stages.
[0012] A further improvement to the above solution is that: the calculation method of the underlying semantic representation model in step 2 is to use the pre-trained language model BERT to initialize the weights of the underlying semantic representation model, and encode the information operation and maintenance professional knowledge data and natural language query data respectively;
[0013] The information operation and maintenance expertise retrieval dataset is represented as: Among them (Q i ,C i ) is the i-th data pair in the information operation and maintenance professional knowledge retrieval data, Q i is the natural language query data in the i-th data pair, C i is the information operation and maintenance professional knowledge data in the i-th data pair;
[0014] The underlying semantic representation model is used to represent the natural language query data Q i Encode and get its corresponding representation vector E i , E i =model(Q i )=[e1,...,e L ]; where L is the natural language query data Q i The length, e i ∈R 1×D , D is the output dimension of the underlying language representation model, and R represents the set of real numbers;
[0015] The underlying semantic representation model is used to represent the information operation and maintenance professional knowledge data C i Encode and get its corresponding representation vector H i , H i =model(C i )=[h1,...,h M ]; where M is the information operation and maintenance professional knowledge data C i The length, h i ∈R 1×D .
[0016] The above solution is further improved in that: the calculation method of the explicit redundancy removal representation learning module in step 2 is:
[0017] For the i-th data pair (Q i ,C i ), using k layers of different one-dimensional dilated convolutional networks to represent the information operation and maintenance professional knowledge data vector H i Perform convolution operation to obtain k different view information operation and maintenance professional knowledge data representation vectors Among them k is the output dimension of the kth dilated convolutional network; connect the outputs of different dilated convolutional networks to obtain a new information operation and maintenance professional knowledge data representation vector S=concat(S1,...,S k )=[s1,...,s M ];
[0018] Use a fully connected network to project the new information operation and maintenance expertise data representation vector S to S′∈R M×N And the importance weight is obtained through Softmax operation, Where N is the output dimension of the fully connected network, W s ∈R N×D and b s ∈R 1×D , are the weights and biases of the fully connected network respectively;
[0019] Will The transpose of H i Multiply them together to get the information operation and maintenance professional knowledge data C i Representation vector after explicit redundancy removal
[0020] The above solution is further improved in that: the calculation method of the implicit redundancy removal representation learning module in step 2 is:
[0021] A subset of the information operation and maintenance expertise retrieval dataset According to its corresponding representation vector, the corresponding representation vector matrix Z∈R is obtained B×M×D , denoted as Z=[H1,...,H B ];
[0022] Compute the covariance matrix representing the vector matrix Z
[0023] Calculate the sum of the squares of the off-diagonal elements in the covariance matrix Cov(Z), introduce the scaling factor D, where D is the dimension of the representation vector matrix Z of the subset, and obtain the covariance regularization term cov(Z).
[0024] The above solution is further improved in that: the method for setting the training parameters in step 3 is that the training of the information operation and maintenance professional knowledge retrieval model is based on the deep learning framework Pytorch, the number of iterations during training is set to 10, and the learning rate is set to 2e -5 , the batch size is set to 64, the maximum sequence lengths of information operation and maintenance professional knowledge data and natural language query data are set to 256 and 128 respectively, and the optimizer is set to AdamW.
[0025] A further improvement of the above scheme is that the method of constructing the loss function in step 3 is to use the cross-modal matching generalized weighted framework to calculate the polynomial weights of positive samples and negative samples respectively, and construct the triple contrast loss L triplet ,
[0026] Among them, G pos is the weight of the positive sample pair, G Neg is the weight of the negative sample pair, and λ is a hyperparameter used to control the distance between the positive sample pair and the negative sample pair in the representation space; given a selected positive pair (H i ,E i + ), its weight G Pos for, Among them S ii is the similarity score, a mis a hyperparameter, m is a positive integer;
[0027] Given a selected negative pair (H i ,E i - ), its weight G Neg for, Among them S ij is the similarity score, b m is a hyperparameter, m is a positive integer;
[0028] The final training target L is the triple contrast loss L triplet and the sum of the covariance regularization term cov(Z), L = L triplet +cov(Z).
[0029] The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning provided by the present invention solves, to a certain extent, the information redundancy problem at the information operation and maintenance professional knowledge representation level that the current existing information operation and maintenance professional knowledge retrieval methods based on deep learning and pre-trained language models pay less attention to. The information operation and maintenance professional knowledge retrieval method proposed in the present invention was experimented on the information operation and maintenance professional knowledge retrieval benchmark dataset. The experimental results show that the information operation and maintenance professional knowledge retrieval method proposed in the present invention is superior to the current advanced retrieval methods. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning proposed in the present invention can be used as a framework for enhancing the performance of pre-trained language models in information operation and maintenance professional knowledge retrieval tasks. The framework does not rely on a specific underlying semantic representation model, and realizes explicit and implicit de-redundant representation learning by constructing an additional loss function. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The present invention will be further described below with reference to the accompanying drawings.
[0031] Figure 1 This is a flowchart of an information operation and maintenance expertise retrieval method based on explicit and implicit de-redundant representation learning according to an embodiment of the present invention.
[0032] Figure 2 This is a flowchart of constructing an information operation and maintenance professional knowledge retrieval dataset according to an embodiment of the present invention.
[0033] Figure 3 This is a flowchart of building an information operation and maintenance professional knowledge retrieval model according to an embodiment of the present invention.
[0034] Figure 4 This is a flowchart of a training information operation and maintenance professional knowledge retrieval model according to an embodiment of the present invention.
[0035] Figure 5 This is an overall structural diagram of the information operation and maintenance professional knowledge retrieval model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] Example: The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning of this embodiment is as follows: Figure 1 As shown, the following steps are included:
[0037] Step 1: Construct an information operation and maintenance expertise retrieval dataset; Figure 2 As shown, information operation and maintenance professional knowledge fragments and natural language questions and answers corresponding to the information operation and maintenance professional knowledge fragments are collected from the open source information operation and maintenance professional knowledge base; the information operation and maintenance professional knowledge fragments are used as information operation and maintenance professional knowledge data, and the natural language questions and answers are used as natural language query data, and the data pairs in the form of <natural language query data, information operation and maintenance professional knowledge data> are constructed, and the data pairs constitute the information operation and maintenance professional knowledge retrieval data set; the operation and maintenance professional knowledge retrieval data set is divided into several subsets, which can be divided into training sets, verification sets, and test sets as needed, and the corresponding information operation and maintenance professional knowledge base files are sorted for use in the verification and testing stages.
[0038] Step 2: Construct an information operation and maintenance professional knowledge retrieval model; Figure 3 As shown in Figure 1, the information operation and maintenance professional knowledge retrieval model consists of the underlying semantic representation model, the explicit de-redundant representation learning module, and the implicit de-redundant representation learning module. The above modules are constructed in sequence. The overall structure of the constructed information operation and maintenance professional knowledge retrieval model is shown in Figure 1. Figure 5 shown.
[0039] The specific calculation method of the underlying semantic representation model is to use the pre-trained language model BERT to initialize the weights of the underlying semantic representation model, and encode the information operation and maintenance professional knowledge data and natural language query data respectively.
[0040] BERT (Bidirectional Encoder Representation from Transformers) is a context-based pre-training model proposed in 2018. It learns the general embedding form of each word through a large amount of corpus and learns context-independent semantic vector representations to achieve modeling of polysemous words.
[0041] The information operation and maintenance expertise retrieval dataset is represented as: Among them (Q i ,C i ) is the i-th data pair in the information operation and maintenance professional knowledge retrieval data, Q i is the natural language query data in the i-th data pair, C i is the information operation and maintenance professional knowledge data in the i-th data pair.
[0042] The underlying semantic representation model is used to represent the natural language query data Q i Encode and get its corresponding representation vector E i , E i =model(Q i )=[e1,...,e L ]; where L is the natural language query data Q i The length, e i ∈R 1×D , D is the output dimension of the underlying language representation model, and R represents the set of real numbers.
[0043] The underlying semantic representation model is used to represent the information operation and maintenance professional knowledge data C i Encode and get its corresponding representation vector H i , H i =model(C i )=[h1,...,h M ]; where M is the information operation and maintenance professional knowledge data C i The length, h i ∈R 1×D .
[0044] The specific calculation method of the explicit redundancy removal representation learning module is as follows: for the i-th data pair (Q i ,C i ), using k layers of different one-dimensional dilated convolutional networks to represent the information operation and maintenance professional knowledge data vector H i Perform convolution operation to obtain k different view information operation and maintenance professional knowledge data representation vectors Among them k is the output dimension of the kth dilated convolutional network. In this embodiment, taking k=7 as an example, seven layers of different one-dimensional dilated convolutional networks are used. The convolution kernel size, dilation rate, and output dimension of each layer of dilated convolutional network are shown in the following table:
[0045] Table 1: Dilated Convolutional Network Parameter Settings
[0046] k 1 2 3 4 5 6 7 <![CDATA[ks k ]]> 1 3 3 3 5 5 5 <![CDATA[dr k ]]> 1 1 2 3 1 2 3 <![CDATA[oc k ]]> 256 128 128 128 128 64 64
[0047] In the table, ks k 、dr k 、oc k , are the convolution kernel size, dilation rate and output dimension of the k-th layer dilated convolutional network respectively.
[0048] Connect the outputs of different dilated convolutional networks to obtain a new information operation and maintenance professional knowledge data representation vector S=concat(S1,...,S k )=[s1,...,s M ].
[0049] Use a fully connected network to project the new information operation and maintenance expertise data representation vector S to S′∈R M×N And the importance weight is obtained through Softmax operation, Where N is the output dimension of the fully connected network, W s ∈R N×D and b s ∈R 1×D , are the weights and biases of the fully connected network, respectively.
[0050] The above results The transpose of H i Multiply them together to get the information operation and maintenance professional knowledge data C i Representation vector after explicit redundancy removal
[0051] The explicit de-redundancy representation learning module transforms the feature matrix dimension R of the information operation and maintenance expertise representation vector H into M×D Compress to R N×D , redundant representations are removed and only the key representations are retained.
[0052] In the specific calculation method of the implicit de-redundant representation learning module, considering that in the information operation and maintenance expertise representation learning, maintaining the diversity of any information operation and maintenance expertise fragment means eliminating the influence of the same token in each information operation and maintenance expertise fragment, therefore, for a subset of the information operation and maintenance expertise retrieval dataset, the subset is usually part of the books in the training set, According to its corresponding representation vector, the corresponding representation vector matrix Z∈R is obtained B ×M×D , denoted as Z=[H1,...,H B ].
[0053] Compute the covariance matrix representing the vector matrix Z
[0054] Calculate the sum of the squares of the off-diagonal elements in the covariance matrix Cov(Z), introduce the scaling factor D, where D is the dimension of the representation vector matrix Z of the subset, and obtain the covariance regularization term cov(Z).
[0055] The implicit de-redundant representation learning module reduces the correlation between the dimensions of the information operation expertise representation vector by reducing the off-diagonal coefficients of the covariance matrix Cov(Z), thereby preventing redundancy in the encoded information.
[0056] Step 3: Train the information operation and maintenance expertise retrieval model; Figure 4As shown, based on the information operation and maintenance professional knowledge retrieval dataset obtained in step 1 and the information operation and maintenance professional knowledge retrieval model obtained in step 2, the training parameters are set, the loss function is constructed, and the information operation and maintenance professional knowledge retrieval model is trained.
[0057] The specific method for setting training parameters is as follows: the training of the information operation and maintenance professional knowledge retrieval model is based on the deep learning framework Pytorch. The number of iterations during training is set to 10 and the learning rate is set to 2e -5 , the batch size is set to 64, the maximum sequence lengths of information operation and maintenance professional knowledge data and natural language query data are set to 256 and 128 respectively, and the optimizer is set to AdamW.
[0058] The specific method of constructing the loss function is to use the cross-modal matching generalized weighted framework to calculate the polynomial weights of positive samples and negative samples respectively, and construct the triple contrast loss L triplet , Among them, G pos is the weight of the positive sample pair, G Neg is the weight of the negative sample pair, and λ is a hyperparameter used to control the distance between the positive sample pair and the negative sample pair in the representation space; given a selected positive pair (H i ,E i + ), its weight G Pos for, Among them S ii is the similarity score, a m is a hyperparameter, and m is a positive integer. Given a selected negative pair (H i ,E i - ), its weight G Neg for, Among them S ij is the similarity score, b m is a hyperparameter, and m is a positive integer.
[0059] The information operation expertise search task requires mutual retrieval between different modalities, so the loss function contains two components. The first term represents the loss incurred in querying information operation expertise, while the second term represents the loss incurred in retrieving information operation expertise from the query.
[0060] The final training target L is the triple contrast loss L triplet and the sum of the covariance regularization term cov(Z), L = L tripet +cov(Z).
[0061] Step 4: Information operation and maintenance professional knowledge retrieval; using the information operation and maintenance professional knowledge retrieval model trained in step 3, calculate the similarity between the information operation and maintenance questions given by the user and the information operation and maintenance professional knowledge data in the knowledge base; select the group with the highest similarity and push it to the user.
[0062] The present invention is not limited to the above embodiments, and any technical solutions formed by equivalent replacements fall within the protection scope required by the present invention.
Claims
1. A method for information operation and maintenance expertise retrieval based on explicit and implicit de-redundant representation learning, characterized in that: The steps include: Step 1: Construct an information operation and maintenance professional knowledge retrieval dataset; collect information operation and maintenance professional knowledge snippets and the natural language questions and answers corresponding to the information operation and maintenance professional knowledge snippets from the information operation and maintenance professional knowledge database; use the information operation and maintenance professional knowledge snippets as information operation and maintenance professional knowledge data and the natural language questions and answers as natural language query data to construct the data pairs of <natural language query data, information operation and maintenance professional knowledge data>, which constitute the information operation and maintenance professional knowledge retrieval dataset; Step 2: Construct an information operation and maintenance expertise retrieval model. The information operation and maintenance expertise retrieval model consists of an underlying semantic representation model, an explicit de-redundant representation learning module, and an implicit de-redundant representation learning module. These modules are constructed in sequence. Step 3: Train the information operation and maintenance expertise retrieval model. Based on the information operation and maintenance expertise retrieval dataset obtained in step 1 and the information operation and maintenance expertise retrieval model obtained in step 2, set training parameters, construct a loss function, and train the information operation and maintenance expertise retrieval model. Step 4: Information operation and maintenance professional knowledge retrieval; Using the information operation and maintenance expertise retrieval model trained in step 3, calculate the similarity between the information operation and maintenance questions given by the user and the information operation and maintenance expertise data in the knowledge base; Select the group with the highest similarity and push it to the user.
2. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 1 is characterized by: In step 1, the operation and maintenance professional knowledge retrieval data set is divided into several subsets, namely a training set, a validation set, and a test set, and the corresponding information operation and maintenance professional knowledge base files are sorted for use in the verification and testing stages.
3. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 1 is characterized by: The calculation method of the underlying semantic representation model in step 2 is to use the pre-trained language model BERT to initialize the weight of the underlying semantic representation model, and encode the information operation and maintenance professional knowledge data and natural language query data respectively; The information operation and maintenance expertise retrieval dataset is represented as: Among them (Q i ,C i ) is the i-th data pair in the information operation and maintenance professional knowledge retrieval data, Q i is the natural language query data in the i-th data pair, C i is the information operation and maintenance professional knowledge data in the i-th data pair; The underlying semantic representation model is used to represent the natural language query data Q i Encode and get its corresponding representation vector E i , E i =model(Q i )=[e1,...,e L ]; where L is the natural language query data Q i The length, e i ∈R 1×D , D is the output dimension of the underlying language representation model, and R represents the set of real numbers; The underlying semantic representation model is used to represent the information operation and maintenance professional knowledge data C i Encode and get its corresponding representation vector H i , H i =model(C i )=[h1,...,h M ]; where M is the information operation and maintenance professional knowledge data C i The length, h i ∈R 1×D .
4. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 3 is characterized by: The calculation method of the explicit redundancy removal representation learning module in step 2 is: For the i-th data pair (Q i ,C i ), using k layers of different one-dimensional dilated convolutional networks to represent the information operation and maintenance professional knowledge data vector H i Perform convolution operation to obtain k different view information operation and maintenance professional knowledge data representation vectors Among them k is the output dimension of the kth dilated convolutional network; connect the outputs of different dilated convolutional networks to obtain a new information operation and maintenance professional knowledge data representation vector S=concat(S1,...,S k )=[s1,...,s M ]; Use a fully connected network to project the new information operation and maintenance expertise data representation vector S to S′∈R M×N And the importance weight is obtained through Softmax operation, Where N is the output dimension of the fully connected network, W s ∈R N×D and b s ∈R 1×D , are the weights and biases of the fully connected network respectively; Will The transpose of H i Multiply them together to get the information operation and maintenance professional knowledge data C i Representation vector after explicit redundancy removal 5. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 4 is characterized by: The calculation method of the implicit redundancy removal representation learning module in step 2 is: A subset of the information operation and maintenance expertise retrieval dataset According to its corresponding representation vector, the corresponding representation vector matrix Z∈R is obtained B×M×D , denoted as Z=[H1,...,H B ]; Compute the covariance matrix representing the vector matrix Z Calculate the sum of the squares of the off-diagonal elements in the covariance matrix Cov(Z), introduce the scaling factor D, where D is the dimension of the representation vector matrix Z of the subset, and obtain the covariance regularization term cov(Z).
6. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 5 is characterized by: The method for setting the training parameters in step 3 is as follows: the training of the information operation and maintenance professional knowledge retrieval model is implemented based on the deep learning framework Pytorch, the number of iterations during training is set to 10, and the learning rate is set to 2e -5 , the batch size is set to 64, the maximum sequence lengths of information operation and maintenance professional knowledge data and natural language query data are set to 256 and 128 respectively, and the optimizer is set to AdamW.
7. The information operation and maintenance professional knowledge retrieval method based on explicit and implicit de-redundant representation learning according to claim 6 is characterized by: The method for constructing the loss function in step 3 is to use the cross-modal matching generalized weighted framework to calculate the polynomial weights of positive samples and negative samples respectively, and construct the triple contrast loss L triplet , Among them, G pos is the weight of the positive sample pair, G Neg is the weight of the negative sample pair, and λ is a hyperparameter used to control the distance between the positive sample pair and the negative sample pair in the representation space; given a selected positive pair (H i ,E i + ), its weight G Pos for, Among them S ii is the similarity score, a m is a hyperparameter, m is a positive integer; Given a selected negative pair (H i ,E i - ), its weight G Neg for, Among them S ij is the similarity score, b m is a hyperparameter, m is a positive integer; The final training target L is the triple contrast loss L triplet and the sum of the covariance regularization term cov(Z), L = L tripet +cov(Z).