Equipment abnormity identification method and device based on reinforcement learning, and electronic equipment
By adjusting the weights of decision tree nodes through reinforcement learning, the problems of low efficiency and high cost in existing technologies are solved, and efficient device anomaly identification is achieved.
Patent Information
- Application Number
- CN202511052806.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-14
AI Technical Summary
Existing supervised learning-based decision trees are inefficient and costly in adjusting node weights, requiring a large number of basic labeled samples.
By using reinforcement learning to determine the temporal difference error information of multimodal state information, the node weights of the initial decision tree are dynamically adjusted to generate the target decision tree to identify anomaly types.
It enables online dynamic fine-tuning of decision tree node weights, improving efficiency, reducing costs, and eliminating the need for retraining and a large number of labeled samples.
Smart Images
Figure CN120951152A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of anomaly detection technology, specifically to a device anomaly identification method, apparatus, and electronic device based on reinforcement learning. Background Technology
[0002] Currently, decision trees are used to diagnose device anomalies. For example, the status information of the device is analyzed by using a decision tree, and the leaf nodes of the decision tree are matched according to the branches of the decision tree. Then, the anomaly type corresponding to the leaf node is taken as the current anomaly type of the device.
[0003] However, current decision trees are typically based on supervised learning, which results in fixed node weights. Adjusting these weights requires retraining based on supervised learning, leading to low efficiency. Furthermore, supervised training demands a large number of labeled samples, resulting in high training costs. Summary of the Invention
[0004] This application provides a device anomaly identification method, apparatus, and electronic device based on reinforcement learning, which can dynamically fine-tune the node weights in the decision tree without retraining or requiring a large number of basic labeled samples, thereby improving efficiency and reducing costs.
[0005] In a first aspect, embodiments of this application provide a device anomaly identification method based on reinforcement learning, the method comprising:
[0006] Acquire multimodal state information for the target device;
[0007] The temporal differential error information corresponding to the multimodal state information is determined by reinforcement learning;
[0008] The target decision tree is obtained by adjusting the node weights of the initial decision tree based on the time-series difference error information, wherein the initial decision tree is obtained through offline pre-training.
[0009] The target decision tree is used to identify the anomaly type information corresponding to the multimodal state information.
[0010] Secondly, embodiments of this application also provide a device anomaly identification apparatus based on reinforcement learning, the apparatus comprising:
[0011] The acquisition module is used to acquire multimodal state information for the target device;
[0012] The determination module is used to determine the temporal difference error information corresponding to the multimodal state information through reinforcement learning;
[0013] An adjustment module is used to adjust the node weights of the initial decision tree according to the time-series difference error information to obtain the target decision tree, wherein the initial decision tree is obtained through offline pre-training;
[0014] The identification module is used to identify the anomaly type information corresponding to the multimodal state information through the target decision tree.
[0015] Optionally, in some embodiments of this application, determining the temporal difference error information corresponding to the multimodal state information through reinforcement learning includes:
[0016] The target action corresponding to the multimodal state information is determined based on the action balancing strategy;
[0017] Execute the target action and generate target reward information;
[0018] The time-series difference error information is calculated based on the target reward information.
[0019] Optionally, in some embodiments of this application, determining the target action corresponding to the multimodal state information according to the action balancing strategy includes:
[0020] The online network of the target dual network module determines the candidate actions corresponding to the multimodal state information and the expected reward information corresponding to each candidate action, wherein the target network of the target dual network module is used to update the parameters of the online network;
[0021] The target action is determined from the candidate actions using the action balancing strategy and the expected reward information.
[0022] Optionally, in some embodiments of this application, the multimodal state information includes at least one of network signal strength, wireless protocol version information, processor architecture information, IO memory information, memory utilization information, processor load information, or diagnostic task queue length information, wherein the diagnostic task queue length information refers to the length information of the diagnostic task queue that includes the multimodal state information.
[0023] The execution of the target action and generation of target reward information includes:
[0024] The execution of the target action yields execution feedback information, which includes at least one of accuracy information, latency information, resource consumption information, or long-tail discovery information. The long-tail discovery information is determined by calculating the Mahalanobis distance between the multimodal state information and the historical state information.
[0025] The target reward information is calculated using a reward function based on each execution feedback information and the corresponding adjustment coefficient.
[0026] Optionally, in some embodiments of this application, obtaining the initial decision tree includes:
[0027] Historical fault information is obtained from the historical fault database;
[0028] The historical fault information is clustered according to the device type to obtain fault cluster information;
[0029] Based on the fault clustering information, an optimization algorithm targeting at least one optimization objective is used to solve for a Pareto-optimal combination of objective weights, wherein the optimization objective includes at least one of accuracy information, resource consumption information, or latency information.
[0030] The initial decision tree is generated based on the target weight combination.
[0031] Optionally, in some embodiments of this application, adjusting the node weights of the initial decision tree based on the temporal difference error information to obtain the target decision tree includes:
[0032] If the target action is discrete, the node weights of the initial decision tree are adjusted according to the proportional adjustment strategy to obtain the target decision tree;
[0033] If the target action is continuous, the node weights of the initial decision tree are adjusted according to the continuous adjustment strategy to obtain the target decision tree.
[0034] Optionally, in some embodiments of this application, the initial decision tree is further provided with interpretable nodes, and the device further includes:
[0035] In the process of adjusting the node weights of the initial decision tree based on the time-series difference error information to obtain the target decision tree, the adjustment log of the node weights is recorded through the interpretable node.
[0036] Optionally, in some embodiments of this application, after identifying the anomaly type information corresponding to the multimodal state information through the target decision tree, the device further includes:
[0037] Record the duration of each node in the target decision tree that has not been triggered;
[0038] The nodes whose non-triggering duration exceeds a preset duration are designated as target nodes;
[0039] The node weight of the target node is attenuated according to the attenuation frequency of the adjustment coefficient.
[0040] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the reinforcement learning-based device anomaly identification method described above.
[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the reinforcement learning-based device anomaly identification method described above.
[0042] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.
[0043] In summary, this application embodiment obtains multimodal state information for a target device, determines the temporal difference error information corresponding to the multimodal state information through reinforcement learning, and adjusts the node weights of the initial decision tree according to the temporal difference error information to obtain the target decision tree. The initial decision tree is obtained through offline pre-training, and the target decision tree is used to identify the anomaly type information corresponding to the multimodal state information.
[0044] In this embodiment, by determining the temporal difference error information and adjusting the node weights of the initial decision tree obtained through offline pre-training based on the temporal difference error information, online dynamic fine-tuning of the decision tree node weights based on reinforcement learning is achieved. Compared with the existing technology that uses supervised learning to train the decision tree, this embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a schematic diagram of a scenario in which a server performs the reinforcement learning-based device anomaly identification method according to an embodiment of this application;
[0047] Figure 2This is a flowchart illustrating the device anomaly identification method based on reinforcement learning provided in an embodiment of this application;
[0048] Figure 3 This is a schematic diagram of the device anomaly identification device based on reinforcement learning provided in an embodiment of this application;
[0049] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0050] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] This application provides a device anomaly identification method, apparatus, electronic device, and computer-readable storage medium based on reinforcement learning. Specifically, this application provides a device anomaly identification apparatus suitable for electronic devices, which include terminal devices or servers. The terminal devices include, but are not limited to, mobile phones, tablets, laptops, smart TVs, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server can be directly or indirectly connected via wired or wireless communication.
[0052] For example, please see Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario in which a server executes the reinforcement learning-based device anomaly identification method according to an embodiment of this application. The process of a terminal device executing the reinforcement learning-based device anomaly identification method can be understood by referring to the server's execution flow. Specifically, the specific execution process of the server executing the reinforcement learning-based device anomaly identification method is as follows:
[0053] Server 101 acquires the multimodal state information of target device 102, determines the temporal differential error information corresponding to the multimodal state information through reinforcement learning, and adjusts the node weights of the initial decision tree according to the temporal differential error information to obtain the target decision tree. The initial decision tree is obtained through offline pre-training, and the target decision tree is used to identify the anomaly type information corresponding to the multimodal state information.
[0054] For example, an initial decision tree is obtained through offline training. Then, when diagnosing anomalies in a target device, time-series differential error information is generated based on the multimodal state information of the target device. The node weights in the initial decision tree are then fine-tuned online based on the time-series differential error information to obtain a target decision tree. The multimodal state information is then identified based on the target decision tree to determine the anomaly type information of the target device.
[0055] In summary, this application embodiment determines the temporal difference error information and adjusts the node weights of the initial decision tree obtained through offline pre-training based on the temporal difference error information, thereby realizing online dynamic fine-tuning of the decision tree node weights based on reinforcement learning. Compared with the existing technology that uses supervised learning to train the decision tree, this application embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs.
[0056] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.
[0057] Please see Figure 2 , Figure 2 This is a flowchart illustrating a reinforcement learning-based device anomaly detection method provided in this application. While the flowchart shows a logical order, in some cases, the steps shown or described may be executed in a different order than that shown in the flowchart. Specifically, the execution entity of this reinforcement learning-based device anomaly detection method includes a server, and the specific flow of the reinforcement learning-based device anomaly detection method is as follows:
[0058] 201. Obtain multimodal state information for the target device.
[0059] The multimodal state information refers to the relevant performance information of the device in its current state. This multimodal state information includes the device's own attribute information as well as resource status information. For example, the attribute information includes network signal strength, wireless protocol version information, processor architecture information, and I / O memory information, while the resource status information includes memory utilization, processor load information, or diagnostic task queue length information. The wireless protocol version information includes Wi-Fi protocol version information or BitTorrent protocol version information, etc. The diagnostic task queue length information refers to the length of the diagnostic task queue containing the multimodal state information. It can be understood that the diagnostic task queue is the queue of tasks currently to be addressed, and the current multimodal state information corresponds to one of the diagnostic tasks in the queue.
[0060] The target device is the device to be diagnosed as abnormal, such as a mobile phone, tablet, or television.
[0061] 202. Determine the temporal difference error information corresponding to the multimodal state information through reinforcement learning.
[0062] Reinforcement learning (RL) is a branch of machine learning that focuses on how an agent learns the optimal policy by interacting with the environment to maximize cumulative reward. Its core idea is trial-and-error learning, optimizing behavior through continuous experimentation and feedback.
[0063] Among them, the temporal difference error (TD-error) is the prediction error in temporal difference learning. In the embodiments of this application, the prediction error corresponding to the multimodal state information is determined by reinforcement learning.
[0064] 203. Adjust the node weights of the initial decision tree according to the time-series difference error information to obtain the target decision tree, wherein the initial decision tree is obtained through offline pre-training.
[0065] For example, the target decision tree with updated node weights can be obtained by using gradient updates based on temporal difference error information.
[0066] The initial decision tree is pre-trained offline. When diagnosing device anomalies based on multimodal state information, the node weights of the initial decision tree are fine-tuned using calculated temporal difference error information. This online fine-tuning method dynamically optimizes and updates the node weights in the decision tree. Compared to traditional decision trees obtained through supervised learning, this reinforcement learning-based online fine-tuning method improves the efficiency of node weight updates.
[0067] 204. Identify the anomaly type information corresponding to the multimodal state information through the target decision tree.
[0068] For example, by using each node in the target decision tree and the node weights corresponding to each node, the abnormality type information corresponding to the current multimodal state information of the target device can be determined, thereby realizing the current abnormality diagnosis of the target device.
[0069] In summary, this application embodiment determines the temporal difference error information and adjusts the node weights of the initial decision tree obtained through offline pre-training based on the temporal difference error information, thereby realizing online dynamic fine-tuning of the decision tree node weights based on reinforcement learning. Compared with the existing technology that uses supervised learning to train the decision tree, this application embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs.
[0070] It should be noted that reinforcement learning constructs the learning process by forming behaviors and feedback through the relationships between states, actions, rewards, and policies. Here, the state is the current observation, the action is the operation selected based on the state, the reward is the immediate feedback signal generated by the operation, and the policy maps the state to the executed operation. Therefore, in this embodiment, the action is determined by the policy, and the temporal difference error information is calculated based on the reward generated by the action execution. That is, optionally, in some embodiments of this application, the step "determining the temporal difference error information corresponding to the multimodal state information through reinforcement learning" includes:
[0071] The target action corresponding to the multimodal state information is determined based on the action balancing strategy;
[0072] Execute the target action and generate target reward information;
[0073] The time-series difference error information is calculated based on the target reward information.
[0074] In reinforcement learning, exploration is the process by which an agent actively tries unknown or suboptimal actions to acquire more environmental information. Its core function is to prevent the policy from getting stuck in local optima, thereby improving long-term gains. Action balancing strategies are used to balance the agent's behavior between knowing the optimal action (exploitation) and trying new actions (exploration). These action balancing strategies include ε-greedy strategies.
[0075] Understandably, the target action is a decision based on an action balancing strategy. This action can be discrete or continuous. Discrete actions include a limited number of selectable actions, such as classification actions for multimedia anomalies, Wi-Fi anomalies, and audio anomalies. Continuous action values refer to actions formed in a continuous form, such as actions related to determining whether audio stuttering is due to volume settings or whether mutating is necessary.
[0076] The target reward information refers to the scalar feedback signal returned after performing the target action (such as score increase or decrease, task completion rate, etc.). Then, the temporal difference error information is calculated using the target reward information. Furthermore, the gradient is calculated using the temporal difference error information (TD-error) and the node weights in the initial decision tree are updated. The temporal difference error (TD-error) is a key indicator measuring the difference between the current value function estimate (e.g., the current state value prediction) and the true value (e.g., the actually observed reward).
[0077] In this embodiment, a target dual-network module based on an online network and a target network is used to output candidate actions corresponding to multimodal state information, and a target action is selected from these candidate actions using an action balancing strategy. Optionally, in some embodiments of this application, the step "determining the target action corresponding to the multimodal state information according to the action balancing strategy" includes:
[0078] The online network of the target dual network module determines the candidate actions corresponding to the multimodal state information and the expected reward information corresponding to each candidate action, wherein the target network of the target dual network module is used to update the parameters of the online network;
[0079] The target action is determined from the candidate actions using the action balancing strategy and the expected reward information.
[0080] The online network includes the Main DQN, which is used to evaluate the current policy. The target network includes the TargetDQN, which is used to fix or stabilize the network. For example, multimodal state information is input into the online network to predict the predicted reward information for all candidate actions, while the target network provides stable target reward information when updating the online network. The reward information includes Q-values, which measure the long-term expected reward of an agent choosing a particular action in a specific state. The predicted reward information is the Q-value output by the online network for each candidate action, while the target reward information is the Q-value generated by the target network.
[0081] Furthermore, in this embodiment, the exploration rate is calculated using an action balancing strategy, and the exploration method or utilization method is determined based on the exploration rate. The exploration rate ε∈[0,1] is a hyperparameter representing the probability that the agent chooses a random exploration action (rather than the currently known optimal action). For example, if the current exploration rate is ε, then unknown or suboptimal actions are attempted through exploration, which may lead to the discovery of a better strategy; if the exploration rate is 1-ε, then the action with the highest predicted reward information is selected through utilization to ensure short-term gains. The exploration rate changes according to a decay strategy, for example, using linear or exponential decay over time. For instance, in the initial stage: a higher exploration rate (e.g., ε is 0.9) encourages exploration; in the later stages of training: the exploration rate is gradually reduced (e.g., ε linearly decays to 0.01), at which point utilization is favored.
[0082] It is understood that the target reward information is calculated based on the feedback after the target action is executed. Optionally, in some embodiments of this application, the step "execute the target action and generate target reward information" includes:
[0083] The execution of the target action yields execution feedback information, which includes at least one of accuracy information, latency information, resource consumption information, or long-tail discovery information. The long-tail discovery information is determined by calculating the Mahalanobis distance between the multimodal state information and the historical state information.
[0084] The target reward information is calculated using a reward function based on each execution feedback information and the corresponding adjustment coefficient.
[0085] Among them, accuracy information includes F1-score, which is the harmonic mean of diagnostic accuracy and recall; latency information refers to the latency of anomaly type diagnosis; resource consumption information mainly refers to the memory resources consumed by anomaly type diagnosis; long-tail discovery information reflects the discovery of new states.
[0086] For example, reward functions include:
[0087]
[0088] Where R represents the target reward information, also known as the reward value, F1-score represents the accuracy information, Latency represents the latency information, Memory represents the resource consumption information, and Novelty represents the long-tail discovery information. 0.6, 0.2, 0.2 and 0.1 are the corresponding adjustment coefficients, and 150ms and 50MB are the default calculation constants.
[0089] The Mahalanobis distance includes the Mahalanobis distance. A first feature vector is obtained by representing the multimodal state information as a feature vector, and a second feature vector is obtained by representing the historical state information (e.g., multimodal state information of historical stages stored in a history database) as a feature vector. The Mahalanobis distance between the first and second feature vectors is then calculated. A larger Mahalanobis distance results in a larger value for the novelty information; for example, the novelty value is equal to the Mahalanobis distance.
[0090] Among them, by considering long-tail discovery information to calculate target reward information, it is possible to detect anomalies in the status information of new devices. For example, when a new HDMI 2.1 interface (not included in the feature library) is detected, the Novelty reward mechanism is triggered in the first 10 diagnoses, automatically increasing the exploration rate, and autonomously generating a dedicated decision path for the interface within 72 hours.
[0091] In this embodiment of the application, the initial decision tree can be generated based on the multimodal state information of historical stages stored in the historical fault database. That is, optionally, in some embodiments of the application, the acquisition of the initial decision tree includes:
[0092] Historical fault information is obtained from the historical fault database;
[0093] The historical fault information is clustered according to the device type to obtain fault cluster information;
[0094] Based on the fault clustering information, an optimization algorithm targeting at least one optimization objective is used to solve for a Pareto-optimal combination of objective weights, wherein the optimization objective includes at least one of accuracy information, resource consumption information, or latency information.
[0095] The initial decision tree is generated based on the target weight combination.
[0096] For example, a decision tree to be optimized is constructed based on fault clustering information, and an initial decision tree is obtained by optimizing the decision tree based on the target weight combination. Here, historical fault information refers to equipment anomaly information from historical periods, including multimodal state information of the equipment in those historical periods and their corresponding anomaly types.
[0097] Among them, the optimization algorithm includes multi-objective optimization algorithm (such as NSGA-II algorithm), which solves the Pareto optimal weight combination by constructing objective function and constraint conditions to obtain the objective weight combination. For example, the objective function includes maximizing accuracy information, minimizing resource consumption information and minimizing latency information, and the constraint conditions include single memory usage less than or equal to 3MB and inference latency (diagnostic latency, i.e., latency information) less than or equal to 200ms.
[0098] In some embodiments of this application, when adjusting the node weights of the initial decision tree, different strategies can be selected to update the node weights based on the different action types of the target action. Specifically, in some embodiments of this application, the step "adjusting the node weights of the initial decision tree according to the temporal difference error information to obtain the target decision tree" includes:
[0099] If the target action is discrete, the node weights of the initial decision tree are adjusted according to the proportional adjustment strategy to obtain the target decision tree;
[0100] If the target action is continuous, the node weights of the initial decision tree are adjusted according to the continuous adjustment strategy to obtain the target decision tree.
[0101] For example, for discrete actions of discrete type, node weights are adjusted using strategies of ±5%, ±10%, and ±15%. For continuous actions of continuous type, the Deep Deterministic Policy Gradient Algorithm (DDPG) is used to implement continuous weight adjustment.
[0102] Furthermore, in this embodiment, interpretable nodes can also be set to record the adjustment log of node weight updates in the decision tree. That is, optionally, in some embodiments of this application, interpretable nodes are also set in the initial decision tree, and the method further includes:
[0103] In the process of adjusting the node weights of the initial decision tree based on the time-series difference error information to obtain the target decision tree, the adjustment log of the node weights is recorded through the interpretable node.
[0104] For example, for node A in the initial decision tree, the corresponding node content is that the wireless network signal strength is less than -70dBm and the initial weight is 0.6. After the gradient update based on the time-series difference error information, the weight is increased by 10%. Then the content of the adjustment log recorded by the node can be interpreted as: Node A - Adjusted weight [+10%] (initial weight 0.6 - 0.66).
[0105] Furthermore, in the embodiments of this application, for nodes in the target decision tree that have not been triggered for a long time, the weight of the node can be gradually reduced through a weight decay strategy. That is, optionally, in some embodiments of this application, after the step "identifying the anomaly type information corresponding to the multimodal state information through the target decision tree", the method further includes:
[0106] Record the duration of each node in the target decision tree that has not been triggered;
[0107] The nodes whose non-triggering duration exceeds a preset duration are designated as target nodes;
[0108] The node weight of the target node is attenuated according to the attenuation frequency of the adjustment coefficient.
[0109] For example, a weight decay can be applied to nodes that have not been triggered for 30 consecutive days, such as a 15% decay per month.
[0110] Furthermore, in this embodiment, the weights of each node in the decision tree are compressed from 32-bit floating-point numbers to 8-bit fixed-point numbers using an error compensation algorithm. This error compensation algorithm includes direct truncation, rounding, random rounding, and dynamic range adjustment.
[0111] In summary, this application embodiment determines the temporal difference error information and adjusts the node weights of the initial decision tree obtained through offline pre-training based on the temporal difference error information, thereby realizing online dynamic fine-tuning of the decision tree node weights based on reinforcement learning. Compared with the existing technology that uses supervised learning to train the decision tree, this application embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs.
[0112] To facilitate better implementation of the reinforcement learning-based device anomaly identification method of this application, this application also provides a reinforcement learning-based device anomaly identification device apparatus based on the aforementioned reinforcement learning-based device anomaly identification method. The meanings of the terms used are the same as in the reinforcement learning-based device anomaly identification method described above, and specific implementation details can be found in the descriptions of the method embodiments.
[0113] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a device anomaly identification device based on reinforcement learning provided in an embodiment of this application. Specifically, the device anomaly identification device based on reinforcement learning can be as follows:
[0114] The acquisition module 301 is used to acquire multimodal status information for the target device;
[0115] The determination module 302 is used to determine the temporal difference error information corresponding to the multimodal state information through reinforcement learning;
[0116] The adjustment module 303 is used to adjust the node weights of the initial decision tree according to the time-series difference error information to obtain the target decision tree, wherein the initial decision tree is obtained through offline pre-training;
[0117] The identification module 304 is used to identify the anomaly type information corresponding to the multimodal state information through the target decision tree.
[0118] Optionally, in some embodiments of this application, determining the temporal difference error information corresponding to the multimodal state information through reinforcement learning includes:
[0119] The target action corresponding to the multimodal state information is determined based on the action balancing strategy;
[0120] Execute the target action and generate target reward information;
[0121] The time-series difference error information is calculated based on the target reward information.
[0122] Optionally, in some embodiments of this application, determining the target action corresponding to the multimodal state information according to the action balancing strategy includes:
[0123] The online network of the target dual network module determines the candidate actions corresponding to the multimodal state information and the expected reward information corresponding to each candidate action, wherein the target network of the target dual network module is used to update the parameters of the online network;
[0124] The target action is determined from the candidate actions using the action balancing strategy and the expected reward information.
[0125] Optionally, in some embodiments of this application, the multimodal state information includes at least one of network signal strength, wireless protocol version information, processor architecture information, IO memory information, memory utilization information, processor load information, or diagnostic task queue length information, wherein the diagnostic task queue length information refers to the length information of the diagnostic task queue that includes the multimodal state information.
[0126] The execution of the target action and generation of target reward information includes:
[0127] The execution of the target action yields execution feedback information, which includes at least one of accuracy information, latency information, resource consumption information, or long-tail discovery information. The long-tail discovery information is determined by calculating the Mahalanobis distance between the multimodal state information and the historical state information.
[0128] The target reward information is calculated using a reward function based on each execution feedback information and the corresponding adjustment coefficient.
[0129] Optionally, in some embodiments of this application, obtaining the initial decision tree includes:
[0130] Historical fault information is obtained from the historical fault database;
[0131] The historical fault information is clustered according to the device type to obtain fault cluster information;
[0132] Based on the fault clustering information, an optimization algorithm targeting at least one optimization objective is used to solve for a Pareto-optimal combination of objective weights, wherein the optimization objective includes at least one of accuracy information, resource consumption information, or latency information.
[0133] The initial decision tree is generated based on the target weight combination.
[0134] Optionally, in some embodiments of this application, adjusting the node weights of the initial decision tree based on the temporal difference error information to obtain the target decision tree includes:
[0135] If the target action is discrete, the node weights of the initial decision tree are adjusted according to the proportional adjustment strategy to obtain the target decision tree;
[0136] If the target action is continuous, the node weights of the initial decision tree are adjusted according to the continuous adjustment strategy to obtain the target decision tree.
[0137] Optionally, in some embodiments of this application, the initial decision tree is further provided with interpretable nodes, and the device further includes:
[0138] In the process of adjusting the node weights of the initial decision tree based on the time-series difference error information to obtain the target decision tree, the adjustment log of the node weights is recorded through the interpretable node.
[0139] Optionally, in some embodiments of this application, after identifying the anomaly type information corresponding to the multimodal state information through the target decision tree, the device further includes:
[0140] Record the duration of each node in the target decision tree that has not been triggered;
[0141] The nodes whose non-triggering duration exceeds a preset duration are designated as target nodes;
[0142] The node weight of the target node is attenuated according to the attenuation frequency of the adjustment coefficient.
[0143] In this embodiment, the acquisition module 301 first acquires multimodal state information for the target device, the determination module 302 determines the temporal difference error information corresponding to the multimodal state information through reinforcement learning, and the adjustment module 303 adjusts the node weights of the initial decision tree according to the temporal difference error information to obtain the target decision tree. The initial decision tree is obtained through offline pre-training, and the identification module 304 identifies the anomaly type information corresponding to the multimodal state information through the target decision tree.
[0144] In this embodiment, the temporal difference error information is determined and the node weights of the initial decision tree obtained by offline pre-training are adjusted based on the temporal difference error information. This realizes online dynamic fine-tuning of the node weights of the decision tree based on reinforcement learning. Compared with the existing technology that uses supervised learning to train the decision tree, this embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs.
[0145] In addition, this application also provides an electronic device, such as Figure 4 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically:
[0146] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:
[0147] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.
[0148] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0149] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0150] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0151] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby implementing the steps in any of the reinforcement learning-based device anomaly identification methods provided in the embodiments of this application.
[0152] This application embodiment obtains multimodal state information for a target device, determines the temporal difference error information corresponding to the multimodal state information through reinforcement learning, and adjusts the node weights of the initial decision tree according to the temporal difference error information to obtain the target decision tree. The initial decision tree is obtained through offline pre-training, and the target decision tree is used to identify the abnormal type information corresponding to the multimodal state information.
[0153] In this embodiment, by determining the temporal difference error information and adjusting the node weights of the initial decision tree obtained through offline pre-training based on the temporal difference error information, online dynamic fine-tuning of the decision tree node weights based on reinforcement learning is achieved. Compared with the existing technology that uses supervised learning to train the decision tree, this embodiment does not require retraining to improve efficiency, nor does it require a large number of basic labeled samples to reduce costs.
[0154] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0155] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0156] To this end, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the reinforcement learning-based device anomaly identification methods provided in this application.
[0157] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0158] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0159] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the reinforcement learning-based device anomaly identification methods provided in this application, the beneficial effects that any of the reinforcement learning-based device anomaly identification methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.
[0160] The foregoing has provided a detailed description of a device anomaly identification method, apparatus, electronic device, and computer-readable storage medium based on reinforcement learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A device anomaly identification method based on reinforcement learning, characterized in that, The method includes: Acquire multimodal state information for the target device; The temporal differential error information corresponding to the multimodal state information is determined by reinforcement learning; The target decision tree is obtained by adjusting the node weights of the initial decision tree based on the time-series difference error information, wherein the initial decision tree is obtained through offline pre-training. The target decision tree is used to identify the anomaly type information corresponding to the multimodal state information.
2. The device anomaly identification method based on reinforcement learning according to claim 1, characterized in that, The step of determining the temporal difference error information corresponding to the multimodal state information through reinforcement learning includes: The target action corresponding to the multimodal state information is determined based on the action balancing strategy; Execute the target action and generate target reward information; The time-series difference error information is calculated based on the target reward information.
3. The device anomaly identification method based on reinforcement learning according to claim 2, characterized in that, Determining the target action corresponding to the multimodal state information based on the action balancing strategy includes: The online network of the target dual network module determines the candidate actions corresponding to the multimodal state information and the expected reward information corresponding to each candidate action, wherein the target network of the target dual network module is used to update the parameters of the online network; The target action is determined from the candidate actions using the action balancing strategy and the expected reward information.
4. The device anomaly identification method based on reinforcement learning according to claim 2, characterized in that, The multimodal state information includes at least one of network signal strength, wireless protocol version information, processor architecture information, IO memory information, memory usage information, processor load information, or diagnostic task queue length information, wherein the diagnostic task queue length information refers to the length information of the diagnostic task queue that includes the multimodal state information. The execution of the target action and generation of target reward information includes: The execution of the target action yields execution feedback information, which includes at least one of accuracy information, latency information, resource consumption information, or long-tail discovery information. The long-tail discovery information is determined by calculating the Mahalanobis distance between the multimodal state information and the historical state information. The target reward information is calculated using a reward function based on each execution feedback information and the corresponding adjustment coefficient.
5. The device anomaly identification method based on reinforcement learning according to claim 1, characterized in that, The acquisition of the initial decision tree includes: Historical fault information is obtained from the historical fault database; The historical fault information is clustered according to the device type to obtain fault cluster information; Based on the fault clustering information, an optimization algorithm targeting at least one optimization objective is used to solve for a Pareto-optimal combination of objective weights, wherein the optimization objective includes at least one of accuracy information, resource consumption information, or latency information. The initial decision tree is generated based on the target weight combination.
6. The device anomaly identification method based on reinforcement learning according to claim 2, characterized in that, The step of adjusting the node weights of the initial decision tree based on the temporal difference error information to obtain the target decision tree includes: If the target action is discrete, the node weights of the initial decision tree are adjusted according to the proportional adjustment strategy to obtain the target decision tree; If the target action is continuous, the node weights of the initial decision tree are adjusted according to the continuous adjustment strategy to obtain the target decision tree.
7. The device anomaly identification method based on reinforcement learning according to claim 1, characterized in that, The initial decision tree also includes interpretable nodes, and the method further includes: In the process of adjusting the node weights of the initial decision tree based on the time-series difference error information to obtain the target decision tree, the adjustment log of the node weights is recorded through the interpretable node.
8. The device anomaly identification method based on reinforcement learning according to claim 1, characterized in that, After identifying the anomaly type information corresponding to the multimodal state information through the target decision tree, the method further includes: Record the duration of each node in the target decision tree that has not been triggered; The nodes whose non-triggering duration exceeds a preset duration are designated as target nodes; The node weight of the target node is attenuated according to the attenuation frequency of the adjustment coefficient.
9. A device anomaly detection device based on reinforcement learning, characterized in that, The device includes: The acquisition module is used to acquire multimodal state information for the target device; The determination module is used to determine the temporal difference error information corresponding to the multimodal state information through reinforcement learning; An adjustment module is used to adjust the node weights of the initial decision tree according to the time-series difference error information to obtain the target decision tree, wherein the initial decision tree is obtained through offline pre-training; The identification module is used to identify the anomaly type information corresponding to the multimodal state information through the target decision tree.
10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the reinforcement learning-based device anomaly identification method as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based device anomaly identification method as described in any one of claims 1-8.