Anomaly detection knowledge graph construction method and system based on reinforcement learning

Abstract

The present invention discloses a method and system for constructing anomaly detection knowledge graphs based on reinforcement learning. The method is as follows: S1, construct anomaly detection ontology; S2, model the external environment as a Markov decision process; S3, generate vectors through word embedding and position embedding, and represent the current state as a cascade of the current sentence vector and the average vector of sentences that have been eliminated from the early stage; S4, for agent behavior, it is expected that the agent will identify and eliminate noisy sentences through a policy network, and the performance change of each step is used to determine the agent's reward; S5, after the action and reward are determined, a policy network is designed, which is used to determine whether to delete a noisy sentence and evaluate each input sentence; the policy network learns useful patterns in each sentence, using a window size of c w , kernel size is c k The graph convolutional network models the policy network π(a|s;θ); S6 pre-trains the policy network; ambiguous sentences are rewarded by the agent through retraining.

Description

Technical Field

[0001] The present invention belongs to the field of anomaly detection technology in artificial intelligence technology, and specifically relates to a high-accuracy, high-speed method and system for constructing anomaly detection knowledge graph based on reinforcement learning. Background Art

[0002] With the in-depth application of knowledge graph technology, knowledge-driven approaches can be used to locate anomalies through knowledge matching and explore their causes. This can integrate multi-source heterogeneous knowledge and provide effective industry solutions for anomaly detection in island photovoltaics. Island photovoltaic industry knowledge has complex hierarchies and relationships. Entity relationship extraction, as a core task in knowledge graph construction, can generate contextual awareness and proactive services. However, when processing large-scale data, entity relationship extraction performance is often hampered by a lack of labeled data. Therefore, the field needs to develop a new reinforcement learning-based anomaly detection knowledge graph construction technology solution to address this issue. Summary of the Invention

[0003] In order to solve the above problem, that is, to solve the problem that the existing entity relationship extraction method is not effective when processing large-scale data due to the lack of labeled data, the present invention provides a high-accuracy and high-speed reinforcement learning-based anomaly detection knowledge graph construction method and system.

[0004] The present invention adopts the following technical solutions:

[0005] The method for constructing anomaly detection knowledge graph based on reinforcement learning includes the following steps:

[0006] Step S100: construct anomaly detection ontology, using event logic to represent domain knowledge, and clearly define the domain and scope of the ontology as anomaly detection;

[0007] Step S200: Model the external environment as a Markov decision process, where the state s (which describes the environmental conditions or situation that the agent is in at a certain moment. The state contains enough information for the agent to decide the best action for the next step) contains the information of the current sentence and the sentences deleted in the previous state;

[0008] Step S300 , generating a vector by word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from the early stage;

[0009] Step S400: For agent behavior, in order to eliminate noisy sentences, the agent is expected to identify and eliminate noisy sentences through a policy network, and the performance change of each step is used to determine the agent's reward;

[0010] Step S500: After the action and reward are determined, a policy network is designed. The policy network is used to determine whether to delete a noise sentence. It evaluates each input sentence to determine whether it expresses the target relationship type and deletes any irrelevant sentences. Therefore, the policy network is similar to a binary relationship classifier and can effectively learn useful patterns in each sentence. w , kernel size is c k The graph convolutional network models the policy network π(a|s; θ) (where π represents the policy function; a represents the action, which is one of the behaviors that the agent can choose in a specific state. In a reinforcement learning environment, the agent interacts with the environment by choosing different actions and updates its strategy based on the feedback from the environment; s is the state s mentioned above; θ is the parameter of the policy network, which controls the shape and behavior of the policy function. During the training process, a set of optimal parameters is found so that the agent's actions in different states can maximize long-term rewards); the action described in this step is the agent's recognition and elimination of noisy statements in step S400, that is, what purpose the agent is expected to achieve, and then the specific action is determined by the policy network, that is, how to achieve the goal. The reward is used to determine the specific action. If the action is closer to achieving the goal, a positive reward is given, otherwise a negative reward is given.

[0011] Step S600: During the training phase of the policy network, pre-training is used to improve the robustness of the modeling. Through pre-training, the model can recognize sentences with obvious noise. For ambiguous sentences, the agent is rewarded through re-training.

[0012] Preferably, in step S100, anomaly detection ontology is constructed through a seven-step creation method of determining domain and scope, reusing existing ontology knowledge, listing important domain terms, determining classes and hierarchical structures, determining class attributes and relationships, defining attribute facts, and creating instances.

[0013] Preferably, in step S100, the object (O), trigger word (T) and state (S) of the anomaly detection event independent variable are defined, the class concept describes the independent variable of the anomaly detection event, the ontology class concept is constructed, and the class relationship describes the definition of the event trigger word relationship and the logical relationship between events. The anomaly detection event independent variable is expressed as:

[0014] e=<O,T,S>

[0015] Among them, O represents the abnormal component, corresponding to the device word; T represents the event trigger element, corresponding to the event trigger description word; S represents the device status, corresponding to the abnormal status word.

[0016] Preferably, in step S300, in order to amplify the main influence of the current sentence information on the action decision, a relatively large weight is allocated to the current sentence vector.

[0017] Preferably, in step S400, the behavior reward is expressed as:

[0018] R i =α(F i -F i-1 )

[0019] Among them, F i is the reward value of the i-th step, and α is used to convert the change of the reward value into the range of -1 to 1.

[0020] Preferably, in step S600, during the retraining phase, the remotely supervised positive dataset is divided into a training positive dataset and a validation positive dataset using a loss function, so that at each iteration, the agent eliminates a noise sample from the training positive dataset according to a random strategy π(a|s;θ). When the loss converges, a cleaned dataset for remotely supervised relationship extraction is obtained. The loss function is expressed as:

[0021]

[0022] Among them, Ω i is the noise removal instance in the i-th iteration, and R is the reward return of the policy network.

[0023] The present invention also discloses a reinforcement learning-based anomaly detection knowledge graph construction system, based on the above method, including the following modules:

[0024] Anomaly detection ontology construction module: Build an anomaly detection ontology, use event logic to represent domain knowledge, and clarify the domain and scope of the ontology as anomaly detection;

[0025] External environment modeling module: The external environment is modeled as a Markov decision process, where the state s contains the information of the current statement and the statements deleted in the previous state;

[0026] Current state representation module: Generates a vector through word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from the early stages;

[0027] Noisy sentence identification and elimination module: For agent behavior, it is expected that the agent will identify and eliminate noisy sentences through the policy network, and use the performance change of each step to determine the agent's reward; the behavior reward is expressed as:

[0028] R i =α(F i -F i-1 )

[0029] Among them, F i is the reward value of step i, and α is used to convert the change of reward value into the range of -1 to 1;

[0030] Policy network design module: After the action and reward are determined, the policy network is designed. The policy network is used to determine whether to delete a noise sentence, evaluate each input sentence to determine whether it expresses the target relationship type, and delete any irrelevant sentences; the policy network learns useful patterns in each sentence, using a window size of c w , kernel size is c k The policy network π(a|s;θ) is modeled by a graph convolutional network.

[0031] Training module: Pre-train the policy network; for ambiguous statements, the agent obtains rewards through retraining.

[0032] This invention provides a high-accuracy, high-speed method and system for constructing anomaly detection knowledge graphs based on reinforcement learning. The anomaly detection knowledge graph constructed by this invention can be used to assist engineers in equipment maintenance. It also performs anomaly analysis and prediction on multi-source data, including data collected by signal sensors, micro-meteorological stations, and intelligent image and video analysis results. This effectively ensures the stable operation of photovoltaic systems, provides auxiliary analysis for anomaly detection, and comprehensively reduces maintenance costs and improves power generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 It is a schematic diagram of the principle of a method for constructing anomaly detection knowledge graph based on reinforcement learning in a preferred embodiment of the present invention.

[0035] Figure 2 This is a block diagram of a system for building an anomaly detection knowledge graph based on reinforcement learning in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the embodiments, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, rather than all of them. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0037] The present invention is further described below with reference to the accompanying drawings and in combination with specific embodiments.

[0038] Figure 1This is a method flow chart of a specific embodiment of the present invention. Figure 1 This embodiment provides a high-accuracy and high-speed method for constructing anomaly detection knowledge graph based on reinforcement learning, which specifically includes the following steps:

[0039] Step S100 constructs an anomaly detection ontology through a seven-step creation method: determining the domain and scope, reusing existing ontology knowledge, listing important domain terms, determining classes and hierarchies, determining class attributes and relationships, defining attribute facts, and creating instances. Domain knowledge is represented using event logic, and the domain and scope of the ontology are clearly defined as anomaly detection. The object (O), trigger word (T), and state (S) of the anomaly detection event independent variable are defined. Class concepts describe the independent variables of anomaly detection events, and ontology class concepts are constructed. Class relationships describe the definition of event trigger word relationships and the logical relationship between events. The anomaly detection event independent variable is represented as:

[0040] e=<O,T,S>

[0041] Among them, O represents abnormal components, corresponding to device words; T represents event trigger elements, corresponding to event trigger description words; S represents device status, corresponding to abnormal status words;

[0042] Step S200 , modeling the external environment as a Markov decision process, where the state s contains information about the current statement and the statements deleted in the previous state;

[0043] In step S300, a vector is generated by word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from earlier stages. Furthermore, a relatively large weight is assigned to the current sentence vector to amplify the primary influence of the current sentence information on the action decision.

[0044] In step S400, for the agent behavior, in order to eliminate noisy sentences, the agent is expected to identify and eliminate noisy sentences through the policy network, and the performance change of each step is used to determine the agent's reward. The behavior reward is expressed as:

[0045] R i =α(F i -F i-1 )

[0046] Among them, F i is the reward value of the i-th step, and α is used to convert the change of the reward value into the range of -1 to 1.

[0047] Step S500: After the action and reward are determined, a policy network is designed. The policy network is used to determine whether to delete a noise sentence. It evaluates each input sentence to determine whether it expresses the target relationship type and deletes any irrelevant sentences. Therefore, the policy network is similar to a binary relationship classifier and can effectively learn useful patterns in each sentence. w , kernel size is c k The policy network π(a|s;θ) is modeled by a graph convolutional network.

[0048] Step S600: During the training phase of the policy network, pre-training is used to improve the robustness of the model. Through pre-training, the model can identify sentences with obvious noise, and the agent receives rewards for ambiguous sentences through retraining.

[0049] During the retraining phase, the remote supervision positive dataset is divided into a training set and a validation set using a loss function. Thus, at each iteration, the agent eliminates a noise sample from the training set according to a random strategy π(a|s;θ). When the loss converges, a clean dataset for remote supervision relationship extraction is obtained. The loss function is expressed as:

[0050]

[0051] Among them, Ω i is the noise removal instance in the i-th iteration, and R is the reward return of the policy network.

[0052] From the above description and result graph, it can be seen that the present invention provides a high-accuracy, high-speed reinforcement learning-based anomaly detection knowledge graph constructed by the anomaly detection knowledge graph that can be used to help engineers maintain equipment, while performing anomaly analysis and prediction on multi-source data such as signal sensors, data collected by micro-meteorological stations, and image and video intelligent analysis results. For example, when the present invention is applied to the intelligent operation and maintenance of island photovoltaic systems, it can effectively predict and diagnose anomalies in photovoltaic power station photovoltaic arrays, accurately locate the anomaly location, and record the anomaly type, anomaly name, number of anomaly occurrences, anomaly cause, and solution for each device, allowing power station operation and maintenance personnel to quickly check equipment anomalies, ensure the safe and efficient operation of the system, and reduce maintenance costs.

[0053] like Figure 2 As shown, this embodiment discloses a reinforcement learning-based anomaly detection knowledge graph construction system, based on the above method embodiment, including the following modules:

[0054] Anomaly detection ontology construction module: Build an anomaly detection ontology, use event logic to represent domain knowledge, and clarify the domain and scope of the ontology as anomaly detection;

[0055] External environment modeling module: The external environment is modeled as a Markov decision process, where the state s contains the information of the current statement and the statements deleted in the previous state;

[0056] Current state representation module: Generates a vector through word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from the early stages;

[0057] Noisy sentence identification and elimination module: For agent behavior, it is expected that the agent will identify and eliminate noisy sentences through the policy network, and use the performance change of each step to determine the agent's reward; the behavior reward is expressed as:

[0058] R i =α(F i -F i-1 )

[0059] Among them, F i is the reward value of step i, and α is used to convert the change of reward value into the range of -1 to 1;

[0060] Policy network design module: After the action and reward are determined, the policy network is designed. The policy network is used to determine whether to delete a noise sentence, evaluate each input sentence to determine whether it expresses the target relationship type, and delete any irrelevant sentences; the policy network learns useful patterns in each sentence, using a window size of c w , kernel size is c k The policy network π(a|s;θ) is modeled by a graph convolutional network.

[0061] Training module: Pre-train the policy network; for ambiguous statements, the agent obtains rewards through retraining.

[0062] For other contents of this embodiment, please refer to the above method embodiment.

[0063] In summary, the present invention provides a high-accuracy and high-speed method and system for constructing anomaly detection knowledge graph based on reinforcement learning, which aims to solve the problem that the entity relationship extraction performance of existing knowledge graph construction methods is often hindered due to the lack of labeled data when processing large-scale data. It includes constructing a domain ontology: using a seven-step construction method to construct an anomaly detection ontology; modeling the external environment as a Markov decision process, generating vectors through word embedding and position embedding, and assigning a larger weight to the current sentence vector; then using reinforcement learning to use the agent as an interpretable interface for entity and environment perception, and used to train noisy sentences to improve overall performance; the present invention can realize the construction of anomaly detection knowledge graph.

[0064] It should be noted that in the description of this invention, the use of terms such as "reinforcement learning," "knowledge graph," "agent," "state," "action," "reward," and "strategy" to refer to specific concepts is intended to clearly express and understand the technical features of this invention, and does not limit its implementation. In actual applications, relevant parameters and methods can be adjusted according to specific scenarios and requirements. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be understood to indicate or imply relative importance.

[0065] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "application", "combination", "integration", etc. should be understood in a broad sense. For example, it can be the direct use of relevant methods and technologies, or modification or expansion based on them; it can be the close combination of multiple methods and technologies to form an overall solution, or the selective adoption of some of them; it can be the embedding of different algorithms or models into a system so that they can work together, or the interaction between different parts can be achieved through interfaces or middleware. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0066] The term "comprise," "comprising," or any other similar term is intended to cover a non-exclusive inclusion such that a process, system, or component that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, system, or component.

[0067] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for constructing anomaly detection knowledge graph based on reinforcement learning, characterized in that: The method comprises the following steps: Step S100: construct anomaly detection ontology, using event logic to represent domain knowledge, and clearly define the domain and scope of the ontology as anomaly detection; Step S200 , modeling the external environment as a Markov decision process, where the state s contains information about the current statement and the statements deleted in the previous state; Step S300 , generating a vector by word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from the early stage; Step S400: For the agent behavior, it is expected that the agent will identify and eliminate noisy statements through the policy network, and the performance change of each step will be used to determine the agent's reward; Step S500: After the action and reward are determined, a policy network is designed. The policy network is used to determine whether to delete a noise sentence, evaluate each input sentence to determine whether it expresses the target relationship type, and delete any irrelevant sentences. The policy network learns useful patterns in each sentence, using a window size of c. w , kernel size is c k The policy network π(a|s;θ) is modeled by a graph convolutional network. π represents the policy function. a represents the action, which is one of the behaviors selected by the agent in a specific state. θ is the parameter of the policy network. Step S600: Pre-train the policy network; for ambiguous sentences, the agent obtains rewards through retraining; In step S400, the behavior reward is expressed as: R i =α(F i -F i-1 ) Among them, F i is the reward value of step i, and α is used to convert the change of reward value into the range of -1 to 1; In step S600, during the retraining phase, the remotely supervised positive dataset is divided into a training positive set and a validation positive set using a loss function. Thus, at each iteration, the agent eliminates a noise sample from the training positive set according to a random strategy π(a|s;θ). When the loss converges, a cleaned dataset for remotely supervised relationship extraction is obtained. The loss function is expressed as: Among them, Ω i is the noise removal instance in the i-th iteration, and R is the reward return of the policy network.

2. The method for constructing anomaly detection knowledge graph based on reinforcement learning according to claim 1, characterized in that: In step S100, anomaly detection ontology is constructed by determining the domain and scope, reusing existing ontology knowledge, listing important domain terms, determining classes and hierarchical structures, determining class attributes and relationships, defining attribute facts, and creating instances.

3. The method for constructing anomaly detection knowledge graph based on reinforcement learning according to claim 1 or 2, characterized in that: In step S100, the object O, trigger word T, and state S of the anomaly detection event independent variable are defined. The class concept describes the independent variable of the anomaly detection event, and the ontology class concept is constructed. The class relationship describes the definition of the event trigger word relationship and the logical relationship between events. The anomaly detection event independent variable is expressed as: e=<O,T,S> Among them, O represents the abnormal component, corresponding to the device word; T represents the event trigger element, corresponding to the event trigger description word; S represents the device status, corresponding to the abnormal status word.

4. A reinforcement learning-based anomaly detection knowledge graph construction system, based on the method according to any one of claims 1 to 3, characterized in that: Includes the following modules: Anomaly detection ontology construction module: Build an anomaly detection ontology, use event logic to represent domain knowledge, and clarify the domain and scope of the ontology as anomaly detection; External environment modeling module: The external environment is modeled as a Markov decision process, where the state s contains the information of the current statement and the statements deleted in the previous state; Current state representation module: Generates a vector through word embedding and position embedding, representing the current state as a concatenation of the current sentence vector and the average vector of sentences eliminated from the early stages; Noisy sentence identification and elimination module: For agent behavior, it is expected that the agent will identify and eliminate noisy sentences through the policy network, and the performance change of each step will be used to determine the agent's reward; the behavior reward is expressed as: R i =α(F i -F i-1 ) Among them, F i is the reward value of step i, and α is used to convert the change of reward value into the range of -1 to 1; Policy network design module: After the action and reward are determined, the policy network is designed. The policy network is used to determine whether to delete a noise sentence, evaluate each input sentence to determine whether it expresses the target relationship type, and delete any irrelevant sentences; the policy network learns useful patterns in each sentence, using a window size of c w , kernel size is c k The policy network π(a|s;θ) is modeled by a graph convolutional network. π represents the policy function. a represents the action, which is one of the behaviors selected by the agent in a specific state. θ is the parameter of the policy network. Training module: Pre-train the policy network; for ambiguous statements, the agent obtains rewards through retraining.

Images