Object Classification Method and Device Based on Distributed Multi-Agent Reinforcement Learning

Abstract

The present disclosure relates to the field of computer technologies, including an object classification method and apparatus based on distributed multi-agent reinforcement learning. At least two training subsets are obtained by partitioning a training set; for each training subset, the corresponding agent performs reinforcement learning on the training subset to obtain local model parameters obtained after this round of training of the agent's reinforcement learning model. The model parameters of different reinforcement learning models before training are all global model parameters; based on the local model parameters corresponding to each reinforcement learning model, the global model parameters obtained in this round of training are determined; when the number of training rounds meets a preset condition, a target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to perform object classification on an input signal to be classified based on the target reinforcement learning model to obtain a classification result; it can ensure the object classification efficiency and improve the ability of the algorithm to process large-scale imbalanced data.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to an object classification method and apparatus based on distributed multi-agent reinforcement learning. Background Art

[0002] With the development of artificial intelligence, object classification models are widely used to classify objects by training neural network models. For example, fault classification of devices and text classification are performed through object classification models. In traditional training methods for object classification models, different types of training data are required to train neural network models. However, in some scenarios, the training data of different classes is usually unbalanced, that is, there is more training data for some classes and less training data for other classes. At this time, the classification performance of the trained object classification model is low. For example, in the scenario of fault classification of devices, the number of training data for general fault classes is small, while the number of training data for normal classes is large.

[0003] To improve the classification performance of the model under unbalanced training data, an object classification method based on agent reinforcement learning is proposed. Traditional object classification methods based on agent reinforcement learning include: obtaining an unbalanced training set; constructing a data environment using the training set; and training a reinforcement learning model through the interaction between the data environment and the agent based on the Deep Q-Leaning Network (DQN) algorithm to obtain an object classification model.

[0004] However, although the object classification method based on agent reinforcement learning can overcome the problem of unbalanced training set data, the classification ability of the object classification model obtained by training with a single agent is limited and may not achieve the desired classification performance. Summary of the Invention

[0005] In view of this, the present disclosure provides an object classification method and apparatus based on distributed multi-agent reinforcement learning, which can achieve more effective recognition under unbalanced data. Each agent is responsible for processing a specific training subset, and the classification result of each sample is determined by its corresponding agent, which can improve the ability of the object classification method to process unbalanced data and the efficiency of processing large-scale unbalanced data.

[0006] According to one aspect of the present disclosure, there is provided an object classification method based on distributed multi-agent reinforcement learning, the method including:

[0007] Obtain a training set, where the training set includes multiple groups of first training data and multiple groups of second training data; among them, the number of the first training data is less than the number of the second training data; each group of first training data includes a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of second training data includes a second sample signal corresponding to a second object classification and a second classification label of the second object classification;

[0008] Divide the training set to obtain at least two training subsets; among them, each training subset includes each piece of first training data and some of the second training data, and the second training data in different training subsets is different;

[0009] For each training subset, use the agent corresponding to the training subset to perform reinforcement learning on the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training; the model parameters of different reinforcement learning models before training are all global model parameters, and the agents corresponding to different training subsets are different;

[0010] Based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training, and the global model parameters obtained in this round of training are used to assign values to the model parameters of each reinforcement learning model before the next round of training;

[0011] When the number of training rounds meets the preset conditions, determine a target reinforcement learning model based on the global model parameters obtained in the last round of training, so as to perform object classification on the input signal to be classified based on the target reinforcement learning model and obtain a classification result.

[0012] In a possible implementation, the second training data is multi-modal data; correspondingly, the dividing the training set to obtain at least two training subsets includes:

[0013] Divide the second training data according to the modality of the second training data to obtain the second training data corresponding to each modality;

[0014] Combine the second training data corresponding to each modality with each group of first training data to obtain a training subset corresponding to each modality.

[0015] In a possible implementation, the multi-modal data includes second sample signals generated by the generation object of the second sample signal under different working conditions, and different working conditions correspond to different modalities.

[0016] In a possible implementation, the for each training subset, use the agent corresponding to the training subset to perform reinforcement learning on the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training includes:

[0017] Create multiple processes, with each process corresponding to one of the agents;

[0018] Through the parallel operation of the corresponding agents by the multiple processes, perform reinforcement learning on the training subset to obtain the local model parameters obtained after this round of training.

[0019] In a possible implementation, for each training subset, performing reinforcement learning on the training subset through the agent corresponding to the training subset to obtain the local model parameters obtained after this round of training by the reinforcement learning model of the agent includes:

[0020] For each training subset, generate the training order of each group of training data in this round of training for the training subset to obtain a data environment; where the training orders corresponding to different rounds of training are different; the training data includes the first training data and the second training data in the training subset;

[0021] Through the data environment, sequentially return the i-th sample data in the i-th group of training data to the agent corresponding to the training subset according to the training order; the agent includes the reinforcement learning model, the reinforcement learning model includes a deep Q-network and a target network, and the network parameters of the initialized deep Q-network and the target network are the same; i is a positive integer taking values sequentially from 1 to n, and n is the total number of training data in the training subset;

[0022] Determine the i-th classification action corresponding to the i-th sample data through the deep Q-network; where the i-th classification action indicates one of the first classification label and the second classification label;

[0023] Through the agent, return the i-th classification action to the data environment for the data environment to determine the i-th reward of the i-th classification action according to the i-th classification action and the classification label corresponding to the sample data, and return the i-th reward and the (i + 1)-th sample data indicated by the training order to the agent;

[0024] Store the i-th experience data in the experience pool of the agent, and update i = i + 1 to trigger the execution of the step of sequentially returning the i-th sample data in the i-th group of training data to the agent corresponding to the training subset according to the training order and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward, and the (i + 1)-th sample data;

[0025] After traversing all groups of training data in the training subset, randomly extract K groups of experience data from the experience pool; K is a positive integer;

[0026] Input each group of empirical data into the deep Q-network and the target network respectively, and update the deep Q-network based on the difference between the first network result of the deep Q-network and the second network result of the target network;

[0027] Every time W groups of empirical data are traversed, synchronize the network parameters of the deep Q-network to the target network until all groups of empirical data are traversed, and obtain the network parameters of the deep Q-network obtained in this round of training; the network parameters of the deep Q-network obtained in this round of training are the local model parameters obtained after this round of training, and W is a positive integer.

[0028] In a possible implementation manner, the determining the global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model includes:

[0029] Obtain the local model parameters corresponding to each reinforcement learning model obtained in this round of training;

[0030] Determine the average value of each local model parameter to obtain the global model parameters obtained in this round of training.

[0031] In a possible implementation manner, the obtaining the local model parameters corresponding to each reinforcement learning model obtained in this round of training includes:

[0032] Read the local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary, where the shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model;

[0033] Or,

[0034] Call a pre-created remote function to obtain the local model parameters obtained by each process training a reinforcement learning model; where different processes are used to train different reinforcement learning models.

[0035] In a possible implementation manner, the target reinforcement learning model is used to perform fault diagnosis on a device; correspondingly, the signal to be classified includes the device signal of the device; the first object classification includes fault signal classification, and the first classification label includes at least one fault classification label for fault signal classification; the second object classification includes normal signal classification, and the second classification label includes a normal classification label.

[0036] According to another aspect of the present disclosure, there is provided an object classification device based on distributed multi-agent reinforcement learning, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above method when executing the instructions stored in the memory.

[0037] According to another aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the above method is implemented.

[0038] According to another aspect of the present disclosure, there is provided a computer program product including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0039] At least two training subsets are obtained by partitioning a training set; wherein each training subset includes respective first training data and some second training data; for each training subset, reinforcement learning is performed on the training subset by an agent corresponding to the training subset to obtain local model parameters obtained by the reinforcement learning model of the agent after this round of training. The model parameters of different reinforcement learning models before training are all global model parameters; based on the local model parameters corresponding to each reinforcement learning model, the global model parameters obtained in this round of training are determined; when the number of training rounds meets a preset condition, a target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to perform object classification on an input signal to be classified based on the target reinforcement learning model to obtain a classification result; the problem of low efficiency of traditional object classification methods can be solved. Since the target reinforcement learning model is obtained through distributed training based on parameter averaging, when performing object classification, only one target reinforcement learning model is used to achieve object classification, without having to go through the reinforcement learning models respectively corresponding to multiple agents for processing, which can ensure the efficiency of object classification. At the same time, the problem that traditional object classification methods have limited classification capabilities in data imbalance and multi-modal scenarios can be solved. By reusing a small number of first training data and having multiple agents perform division of labor learning on the second training data, the ability of the algorithm to process large-scale imbalanced data can be improved.

[0040] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are included in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and are used to explain the principles of the present disclosure.

[0042] Figure 1 The flowchart showing an object classification method based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure;

[0043] Figure 2 The schematic diagram showing the state movement of an agent at each time step in one round of training according to an embodiment of the present disclosure;

[0044] Figure 3 A schematic diagram showing a deep learning process according to an embodiment of the present disclosure;

[0045] Figure 4 A schematic diagram showing the code implementation of a parameter averaging algorithm according to an embodiment of the present disclosure;

[0046] Figure 5 A schematic diagram showing a fault diagnosis method based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram showing a bearing test bench device according to an embodiment of the present disclosure;

[0048] Figure 7 A block diagram showing an object classification device based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure;

[0049] Figure 8 A block diagram showing an object classification device based on distributed multi-agent reinforcement learning according to another embodiment of the present disclosure. Detailed implementation manners

[0050] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0051] The term "exemplary" used herein means "serving as an example, embodiment, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0052] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0053] First, several terms related to the present application are introduced.

[0054] Reinforcement Learning (RL) is a machine learning method that achieves decision-making and behavior optimization through the interaction and learning between an agent and the environment. In reinforcement learning, the agent explores the environment by trying different actions, adjusts its policy based on the feedback (reward) from the environment to maximize the long-term cumulative reward value. Among them, the agent and the environment are two core components in reinforcement learning.

[0055] Agent: It is the decision-maker in the reinforcement learning system. It achieves a certain goal by perceiving the environment and executing actions. The agent can be a robot, a software program, or any entity that can interact with the environment and take actions.

[0056] Environment: It is the external world where the agent is located, including everything that the agent can perceive and affect. The environment provides the agent with state information and gives corresponding feedback based on the agent's actions. In this application, the agent is a software program, the environment is constructed based on training data, the action indicates one of multiple classification labels in the training data. At this time, the agent interacts with the environment by deciding the classification label, and the environment provides reward feedback to the agent according to this classification label.

[0057] Interaction relationship: The interaction between the agent and the environment is carried out through a loop process. At each time step, the agent perceives the current state (s), for example, the current state is the sample signal in the training data feedback by the environment, selects an action (a) based on the adopted policy (π), for example, the action indicates a classification decided by the agent according to this sample signal; then executes this action, and the environment updates the state accordingly and gives the agent a reward (r), for example, updates the current state to the sample signal in the next set of training data and gives the reward for the classification (or action) corresponding to the current state. This process repeats continuously, enabling the agent to learn the optimal policy through trial and error to maximize the cumulative reward.

[0058] In reinforcement learning, state, action, and reward are the basic elements describing the interaction between the agent and the environment.

[0059] State (State, s): The state is a description of the environment at a certain moment, which can be a comprehensive description of the environment or a set of certain key features. The state provides the information required for the agent to make decisions. The representation of the state can be discrete (such as the position on the chessboard) or continuous (such as the position information of the robot). In this application, the state is the sample signal in each group of training data.

[0060] Action (a): An action is an operation that an agent can perform in a given state. The action space can be a finite discrete set of actions or an infinite continuous set of actions. The agent affects the environment by choosing an action and transitions to a new state. In this application, an action indicates one of the classification labels corresponding to each sample signal.

[0061] Reward (r): A reward is a feedback signal from the environment to the action performed by the agent, used to measure the quality of the action in the current state. The reward function defines the immediate return for each state-action pair. The goal of the agent is to maximize its cumulative reward by choosing appropriate actions.

[0062] The goal of reinforcement learning is to find a policy that enables the agent to obtain the maximum cumulative reward in its interaction with the environment.

[0063] Policy (π): It is a rule or function for the agent to choose an action in each state, which can be deterministic (choosing a fixed action in each state) or stochastic (choosing different actions with a certain probability in each state). The policy can be expressed as π(s) = a, indicating choosing action a in state s, or π(a|s), indicating the probability of choosing action a in state s.

[0064] Value Function (V): It is used to evaluate the expected cumulative reward of a certain state when executing a specific policy. The state value function V(s) represents the expected cumulative reward that can be obtained starting from state s and following policy π.

[0065] Q - Function (Q): The Q - function (state - action value function) evaluates the expected cumulative reward that can be obtained by performing a specific action in a specific state. The Q - value function Q(s, a) represents the expected cumulative reward obtained by performing action a in state s and then following policy π.

[0066] The value function and the Q - function are key tools in reinforcement learning, used to evaluate and optimize policies. By learning and updating these functions, the agent can gradually improve its policy to achieve optimal decision - making.

[0067] Reinforcement learning encompasses a variety of core algorithms, each designed to solve different types of decision-making problems. These algorithms include value iteration, policy iteration, Q-learning, Deep Q-Leaning Network (DQN), policy gradient methods, and actor-critic methods, etc. They handle the interaction between the agent and the environment in different ways and maximize the expected cumulative reward by optimizing the policy. Among them, DQN can break data correlation and improve the model performance under unbalanced training data by storing the agent's experience data through an experience replay mechanism and randomly sampling mini-batch data for training. Based on this, in this application, the interaction between the agent and the environment is taken as an example in the DQN manner for illustration.

[0068] Next, the object classification method based on distributed multi-agent reinforcement learning provided in this application will be described in detail. The application scenarios of the object classification method provided in this application include but are not limited to: fault diagnosis scenarios (i.e., classifying whether the signal to be classified is a fault signal and the fault type), text classification scenarios (i.e., classifying which word the signal to be classified corresponds to), etc. This embodiment does not limit the application scenarios of the object classification method.

[0069] Taking the fault diagnosis scenario as an example, fault diagnosis is the core content of system operation and maintenance and a key step to ensure the normal operation of the system. Through effective fault diagnosis, the root cause of the problem can be quickly located and the correct measures can be taken for repair, thereby minimizing the system downtime and improving production efficiency. Therefore, timely and accurate fault diagnosis is crucial for the normal operation of industrial systems.

[0070] Traditional fault diagnosis methods mainly rely on expert knowledge and rules, and fault models and rule bases need to be formulated according to experience. However, these models can only describe the signal characteristics of several well-defined fault types, but the actual natural faults that occur are often more complex, and traditional methods cannot meet the requirements of large-scale and complex fault diagnosis. To overcome the limitations of traditional methods, data-driven fault diagnosis methods have received extensive attention in recent years. These methods use big data and machine learning technologies to automatically learn fault characteristics and patterns through the analysis and mining of a large amount of historical fault data and perform fault diagnosis based on this. In the current rapidly developing information technology environment, adopting data-driven intelligent fault diagnosis methods has become an essential choice.

[0071] With the rapid development of artificial intelligence technology, the use of artificial intelligence technology to achieve intelligent fault diagnosis makes the fault diagnosis process more intelligent and automated. Recently, deep neural networks such as Deep Autoencoder (DAE) and Deep Convolutional Neural Network (DCNN) have been widely used to build end-to-end intelligent diagnosis models, reducing the dependence on manual experience and expert knowledge, and greatly promoting the development of intelligent fault diagnosis. However, although deep learning models have made some progress in the field of industrial process fault diagnosis, they still face many difficulties and challenges. In actual engineering scenarios, machines usually work in a normal state, which means that the available fault data is limited, resulting in the problem of unbalanced fault samples. Unbalanced fault samples refer to the uneven distribution of the number of samples of different categories in the dataset, that is, the number of samples of the fault category is small, while the number of samples of the normal category is large. Intelligent fault diagnosis under unbalanced data refers to using limited machine fault samples to establish a fault diagnosis model to achieve accurate fault identification. Traditional fault diagnosis methods for unbalanced data include, but are not limited to: data augmentation-based strategies, feature learning-based strategies, and classifier design-based strategies. Data augmentation-based strategies can improve the performance of the diagnosis model by augmenting training data. Feature learning-based strategies can improve the diagnosis performance by designing a regularized neural network or feature adaptation to extract features from limited fault data. Classifier design-based strategies can achieve high-precision fault diagnosis by constructing a classifier suitable for unbalanced data. In addition, in the case where the number of minority samples is extremely small, some knowledge-driven methods with limited sample data are proposed, such as meta-learning and zero-shot learning. At the same time, a general unbalanced sample selection strategy DiagSelect based on deep reinforcement learning is also proposed to solve the problem of data imbalance in the field of fault diagnosis. However, the fault classification ability of the fault diagnosis model obtained by training a single agent is limited and may not achieve the desired classification performance.

[0072] In a possible implementation, the data processing efficiency and classification performance can be improved by introducing a multi-agent system. However, due to the huge number of normal samples, it may be impossible for agents to share normal samples. Therefore, how to achieve accurate and efficient fault diagnosis in the case of data imbalance and the existence of multi-modal normal samples has become an important research issue.

[0073] In addition, after introducing the multi-agent system, multiple reinforcement learning models corresponding to multiple agents will be obtained after training. At this time, in the actual fault diagnosis process, each signal to be classified needs to be respectively processed by the reinforcement learning models corresponding to multiple agents for decision-making, and then the decision results are fused to obtain the final diagnosis result. At this time, the problem of low diagnosis efficiency will occur.

[0074] For the object classification methods in other scenarios, the above problems of low classification efficiency, data imbalance, and limited classification ability in multi-modal scenarios also exist, and these will not be elaborated one by one in this embodiment.

[0075] Based on the above technical problems, in this application, multiple agents are used to perform reinforcement learning on at least two training subsets in the training set respectively, that is, each agent is responsible for processing a specific training subset, and the classification result of the sample signal in each group of training data is determined by its corresponding agent. An object classification method based on distributed multi-agent reinforcement learning is established. This method can improve the ability of the algorithm to process large-scale imbalanced data by reusing a small number of first training data and having multiple agents perform division-of-labor learning on the second training data. At the same time, based on distributed training with parameter averaging, a target reinforcement learning model is obtained. When performing object classification, only one target reinforcement learning model is used to achieve object classification, without the need to be processed by the reinforcement learning models corresponding to multiple agents respectively, which can ensure the object classification efficiency.

[0076] Figure 1 The flowchart of an object classification method based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure is shown. In this embodiment, it is illustrated by taking this method as an example for an electronic device with processing capabilities such as a user terminal or a server. The user terminal includes, but is not limited to: a computer, a tablet computer, a mobile phone, etc. The implementation manner of the electronic device is not limited in this embodiment. As Figure 1 shown, the method includes:

[0077] Step 101, obtain a training set.

[0078] The training set includes multiple groups of training data, and the multiple groups of training data include multiple groups of first training data and multiple groups of second training data. Among them, the number of the first training data is less than the number of the second training data; that is, the training set is an imbalanced data set, and the number of the first training data is small. Each group of training data includes a sample signal and the classification label corresponding to the sample signal. For example: each group of first training data includes a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of second training data includes a second sample signal corresponding to a second object classification and a second classification label of the second object classification.

[0079] Taking the object classification scenario as an example of the fault diagnosis scenario, the first training data can be fault data. Correspondingly, the first training data includes a fault signal and a fault classification label corresponding to the fault signal. The fault classification label can be one or at least two. In the case where the fault classification label includes at least two, different fault classification labels are used to represent different fault classifications. The second training data can be normal data.

[0080] The training set can be obtained by dividing the original dataset according to a certain proportion. For example, as shown in Table 1 below, each group of data in the original dataset includes a sample signal and a classification label corresponding to the sample signal. Among them, the classification labels include fault classification labels 1, 2, 3, and a normal classification label 0; the fault classification label 1 represents a fault classification of IR007, and the number of sample data with the fault classification label 1 is 1250; the fault classification label 2 represents a fault classification of B007, and the number of sample data with the fault classification label 2 is 1250; the fault classification label 3 represents a fault classification of OR007, and the number of sample data with the fault classification label 3 is 1250. The sample signals with the normal classification label 0 include three modes, namely: Normal - 0hp (indicating a working condition of 0 horsepower), Normal - 1hp (indicating a working condition of 1 horsepower), and Normal - 2hp (indicating a working condition of 2 horsepower). Among them, the number of sample signals in each mode is 8750. Thus, the original dataset includes 30000 groups of data.

[0081] Table 1:

[0082] Label Category Mode Number of Samples 0 Normal Normal-0hp 8750 0 Normal Normal-1hp 8750 0 Normal Normal-2hp 8750 1 IR007 None 1250 2 B007 None 1250 3 OR007 None 1250

[0083] If the 30000 groups of data in the original dataset are divided into a training set and a test set according to a ratio of 8:2, 24000 groups of training data for the training set and 6000 groups of test data for the test set can be obtained. Among them, the training set is as shown in Table 2 below. At this time, the training data includes the first training data with fault classification labels 1, 2, and 3 respectively. The fault classification label 1 represents a fault classification of IR007, and the number of the first sample signals with the fault classification label 1 is 1000; the fault classification label 2 represents a fault classification of B007, and the number of the first sample signals with the fault classification label 2 is 1000; the fault classification label 3 represents a fault classification of OR007, and the number of the first sample signals with the fault classification label 3 is 1000. The training data also includes the second training data with a normal classification label of 0. The second training data includes three modes, namely: Normal - 0hp (indicating a working condition of 0 horsepower), Normal - 1hp (indicating a working condition of 1 horsepower), and Normal - 2hp (indicating a working condition of 2 horsepower). Among them, the number of the second sample signals in each mode is 7000.

[0084] Table 2

[0085] Label Category Mode Number of Samples 0 Normal Normal-0hp 7000 0 Normal Normal-1hp 7000 0 Normal Normal-2hp 7000 1 IR007 None 1000 2 B007 None 1000 3 OR007 None 1000

[0086] In other embodiments, the entire original data set may also be used as the training set, and the test set required during the training process is obtained by additional collection. This embodiment does not limit the acquisition method of the training set. The original data set can be obtained by signal collection in an actual classification scenario to be classified. For example, in a fault diagnosis scenario, sample signals are collected and labeled; or, it can also be sent by other devices; this embodiment does not limit the acquisition method of the original data set.

[0087] Step 102: Divide the training set to obtain at least two training subsets.

[0088] Among them, each training subset includes each piece of first training data and part of the second training data, and the second training data in different training subsets is different.

[0089] In this embodiment, by dividing each piece of second training data into different training subsets and reusing the first training data in different training subsets, the subsequent intelligent agent can not only perform division of labor learning on a relatively large-scale second training data to ensure the efficiency of reinforcement learning, but also make full use of the relatively small-scale first training data to ensure the performance of reinforcement learning.

[0090] In an example, the second training data is multi-modal data; correspondingly, dividing the training set to obtain at least two training subsets includes: dividing the second training data according to the modality of the second training data to obtain the second training data corresponding to each modality; merging the second training data corresponding to each modality with each group of first training data to obtain the training subset corresponding to each modality.

[0091] Optionally, the multi-modal data includes second sample signals generated by the generation object of the second sample signal under different working conditions, and different working conditions correspond to different modalities. Among them, the generation object of the second sample signal can be the device that generates the second sample signal, or it can also be the device that collects the second sample signal. This embodiment does not limit the implementation manner of the generation object. The working condition refers to the working state of the generation object when generating the second sample signal. The same generation device can work under different working conditions. At this time, the generated second sample signals may be different. For example: the working condition is the rotational speed of the engine, and the second sample signals generated at different rotational speeds may be different.

[0092] Correspondingly, dividing the second training data according to the modality of the second training data to obtain the second training data corresponding to each modality includes: dividing the training data where the second sample signals of the same modality are located into the same training subset.

[0093] For example: in the training set shown in Table 2 above, after dividing the training set according to modalities, the obtained training subsets are: Training Subset 1, Training Subset 2, and Training Subset 3. Among them, Training Subset 1 includes the second training data with the modality of Normal-0hp and all the first training data; Training Subset 2 includes the second training data with the modality of Normal-1hp and all the first training data; Training Subset 3 includes the second training data with the modality of Normal-2hp and all the first training data.

[0094] Table 3:

[0095] Training Subset 1 Training Subset 2 Training Subset 3 Normal-0hp 7000 0 0 Normal-1hp 0 7000 0 Normal-2hp 0 0 7000 IR007 1000 1000 1000 B007 1000 1000 1000 OR007 1000 1000 1000

[0096] In other embodiments, the multi-modal data may also be the second sample signal of the multi-data domain obtained after converting the second sample signal of the first data domain into the second data domain, where different data domains correspond to different modalities. For example: converting the second sample signal in the time domain into the first time-frequency domain data through the S transform, and / or converting the second sample signal in the time domain into the second time-frequency domain data through the wavelet transform, etc. At this time, the second sample signal in the time domain, the second sample signal of the first time-frequency domain data, and the second sample signal of the second time-frequency domain data are respectively divided into different training subsets. At this time, the first sample signal of the first data domain may also be subjected to the same data domain transformation to be converted into the second data domain, and the first sample signals of all data domains are reused by different training subsets; or, the first sample signal may not be subjected to data domain conversion, and the first sample signals of the first data domain are reused by different training subsets. This embodiment does not limit the implementation manner of the multi-modal data.

[0097] In other embodiments, the second training data may also be randomly and evenly divided, or divided into different training subsets according to a ratio. This embodiment does not limit the division method of the training set.

[0098] Step 103, for each training subset, perform reinforcement learning on the training subset through the agent corresponding to the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training.

[0099] The model parameters of different reinforcement learning models before training are all global model parameters, and the agents corresponding to different training subsets are different. Different agents refer to: different entity software programs that interact with the data environment, and this entity software program may be the process running the agent.

[0100] Optionally, each agent performs reinforcement learning on the training subsets in parallel to improve the training efficiency of the reinforcement learning model. At this time, for each training subset, the corresponding agent of the training subset performs reinforcement learning on the training subset to obtain the local model parameters obtained by the agent's reinforcement learning model after this round of training, including: creating multiple processes, with one process corresponding to one agent; using the multiple processes to run the corresponding agents in parallel to perform reinforcement learning on the training subset to obtain the local model parameters obtained after this round of training.

[0101] In one example, creating multiple processes includes: using the spawn method in the torch.multiprocessing module in Python to create multiple processes.

[0102] In Python, the torch.multiprocessing module provides the function of multiple processes and can be used for parallel computing. The spawn method is a startup method in the torch.multiprocessing module, and this startup method can ensure that the code in different created processes can run independently.

[0103] In another example, creating multiple processes includes: initializing a single-machine multi-process Ray cluster through the ray.init() function in Ray; defining a remote function through the @ray.remote decorator in Ray. Among them, Ray is an open-source distributed framework that supports multiple processes and distributed computing, and the ray.init() function in Ray is used to start multiple local processes and allow multiple tasks to be executed concurrently. The @ray.remote decorator is used to declare a function or class in a remotely callable form so that it can be executed in parallel on different nodes of the Ray cluster.

[0104] In other embodiments, the creation method of single-machine multi-processes can also be other methods, and this embodiment does not limit the creation method of multiple processes.

[0105] In other embodiments, the corresponding agent of the training subset performing reinforcement learning on the training subset may not be achieved through parallel training, but may be trained sequentially according to a preset training order, or part of them are trained sequentially and the other part is trained in parallel. This embodiment does not limit the method by which each agent performs reinforcement learning on the corresponding training subset.

[0106] Specifically, assume that the training set S is divided into 3 training subsets, namely: S1, S2, and S3. At this time, the training set can be represented by the following formula:

[0107] S = {S1, S2, S3}

[0108] Among them, S j represents the training subset corresponding to the agent running in the j-th process, where j is less than or equal to 3. For each training subset S j , the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training, including the following steps 1031-1038:

[0109] Step 1031, for each training subset, generate the training order of each group of training data in this round of training to obtain a data environment; where the training data includes the first training data and the second training data in the training subset.

[0110] The training orders corresponding to different rounds of training are different. Each training subset corresponds to a data environment. Optionally, before each round of training, the data environment shuffles the order between different groups of training data in the training subset, and uses the shuffled order as the training data, and returns the sample signal in each group of training data to the agent as the current state.

[0111] Step 1032, the data environment sequentially returns the i-th sample data in the i-th group of training data to the agent corresponding to the training subset according to the training order; the agent includes a reinforcement learning model, and the reinforcement learning model includes a deep Q network and a target network. Initialize the network parameters of the deep Q network and the target network to be the same; i is a positive integer that takes values from 1 to n in sequence, and n is the total number of training data in the training subset.

[0112] In an example, the network structures of the deep Q networks in the reinforcement learning models of different agents are the same, that is, the same input layer, hidden layer, and output layer are set. In this way, it can be ensured that the model structure remains consistent in each process in a distributed environment, so that the training efficiency is basically the same. In addition, since different agents use different training subsets, the model parameters of the deep Q networks in different agents may be different.

[0113] In other implementations, the network structures of the deep Q networks in different agents can also be different. This embodiment does not limit the model setting method of the deep Q networks in different agents.

[0114] In addition, the network structures of the deep Q network and the target network in the same agent are the same, but the model parameters can be different.

[0115] For example, the network structures of the deep Q-network and the target network in the reinforcement learning models of different agents include: two linear layers, which are respectively set to have 128 neurons and 32 neurons. The number of neurons in the input layer is the dimension of the sample signal, such as 1200, and the number of neurons in the output layer is 4, which respectively represent the normal class and 3 fault classes, for a total of four classifications. In other embodiments, the network structures of the deep Q-network and the target network can also be other implementation manners, and this embodiment does not limit the implementation manners of the deep Q-network and the target network.

[0116] In this step, the data environment inputs the i-th sample data in the i-th group of training data as the current state into the deep Q-network in the agent. Initialize, i = 1, that is, in the first time step of this round of training, the first sample data x1 in the first group of training data is input as the current state (or initial state) s1 into the deep Q-network in the agent.

[0117] Step 1033, determine the i-th classification action corresponding to the i-th sample data through the deep Q-network; wherein, the i-th classification action indicates one of the first classification label and the second classification label.

[0118] In this embodiment, the action (i.e., the i-th classification action) performed by the agent based on the current state (i.e., the i-th sample data) is used to indicate one of the first classification label and the second classification label. Schematically, each classification label in the training set is pre-mapped to an action set A = {0, 1,..., z}. Wherein, z is the total number of classification labels - 1. The action set A includes multiple different values, each value is used to indicate a classification label, and different values indicate different classification labels. For example: the value of z is 9, 0 represents the first classification label of the normal classification, 1 to 9 all represent the second classification labels of the fault classification, and different values among 1 to 9 represent the fault classification labels of different fault types. In other implementations, the values in A can also be other implementation manners, for example: values that do not include 0, etc., and this embodiment does not limit the implementation manner of the action set A.

[0119] In an example, in order to avoid the problem of falling into a local optimal solution when only using the deep Q-network to determine the i-th classification action, the ε-greedy strategy can be combined to determine the i-th classification action. The deep Q-network combined with the ε-greedy strategy to determine the i-th classification action can be expressed by the following formula:

[0120]

[0121] where, a iDenote the $i$-th classification action; $a$ represents a classification action randomly determined from the action set $A$; randomnumber is a random number generated by the data environment within the range of $[0, 1]$; $\varepsilon$ represents a preset threshold in the data environment, where $0 < \varepsilon < 1$, and is used to measure the exploration and exploitation degree of the agent in the learning process; max a $Q(s, a; \theta)$ represents the maximum value among the Q-values corresponding to each classification action obtained by the deep Q-network for the $i$-th sample data, max a In $Q(s, a; \theta)$, $a$ represents different classification actions, $s$ represents the current state (i.e., the $i$-th sample data), $\theta$ is the current model parameter of the deep Q-network, and $Q$ represents the Q-values corresponding to different classification actions output by the deep Q-network. The above indicates that if the random number generated by the data environment within the range of $[0, 1]$ is less than $\varepsilon$, then a random $i$-th classification action is selected, and this $i$-th classification action is different from max a $Q(s, a; \theta)$; if the random number generated by the data environment within the range of $[0, 1]$ is greater than or equal to $\varepsilon$, then max a $Q(s, a; \theta)$ is selected as the $i$-th classification action.

[0122] Step 1034: The agent returns the $i$-th classification action to the data environment, so that the data environment determines the $i$-th reward of the $i$-th classification action according to the classification label corresponding to the $i$-th classification action and the sample data, and returns the $i$-th reward and the $(i + 1)$-th sample data indicated by the training order to the agent.

[0123] In one example, the data environment determines the $i$-th reward of the $i$-th classification action through a reward function. Schematically, this reward function is represented by the following formula:

[0124]

[0125] Among them, $r(s, a, s')$ represents the reward in the case of the current state $s$, the next state being $s'$, and the classification action corresponding to the current state being $a$; if the classification action is the same as the classification label corresponding to the current state $s$ (i.e., right), then $r(s, a, s')$ is 1; if the classification action is different from the classification label corresponding to the current state $s$ (i.e., wrong), then $r(s, a, s')$ is -1.

[0126] Step 1035: The agent stores the $i$-th experience data in the experience pool, and updates $i = i + 1$ to trigger the execution of the step of returning the $i$-th sample data in the $i$-th set of training data to the corresponding agent of the training subset in sequence according to the training order by the data environment and subsequent steps; the $i$-th experience data includes the $i$-th sample data, the $i$-th classification action, the $i$-th reward, and the $(i + 1)$-th sample data.

[0127] Each group of the $i$-th experience data stored in the experience pool can be represented by the following formula:

[0128] B i =(s i , a i , r i , s i+1 );

[0129] Among them, B i represents the i-th empirical data, s i represents the i-th sample data, a i represents the i-th classification action, r i represents the i-th reward, s i+1 represents the (i + 1)-th sample data.

[0130] Assume that the state transition diagram of the agent at each time step in this round of training is as shown in Figure 2 . The data environment returns states s1, s2,... s n to the agent in sequence according to the training order. After the deep Q-network calculates each state respectively, the classification action corresponding to this state is obtained. Figure 2 In, taking z as 4 and a total of 5 classifications as an example for illustration, each classification action corresponds to a reward r.

[0131] Step 1036, after traversing all groups of training data in the training subset, randomly extract K groups of empirical data from the experience pool. Among them, K is a positive integer.

[0132] Step 1037, input each group of empirical data into the deep Q-network and the target network respectively, and update the deep Q-network based on the difference between the first network result of the deep Q-network and the second network result of the target network.

[0133] Specifically, use the stochastic gradient descent method to train the weight and bias value parameters of the deep Q-network to minimize the difference between the first network result and the second network result of the target network, so as to maximize the Q function value and obtain the optimal strategy of the deep Q-network. Among them, the difference between the first network result and the second network result of the target network is calculated through a preset loss function, and this loss function can be a mean square error cost function loss function established in combination with the reward. In other embodiments, the loss function can also be other implementation methods, and the implementation method of the loss function is not limited in this embodiment.

[0134] Step 1038, every time W groups of empirical data are traversed, synchronize the network parameters of the deep Q-network to the target network until all groups of empirical data are traversed, and obtain the network parameters of the deep Q-network obtained in this round of training; the network parameters of the deep Q-network obtained in this round of training are the local model parameters obtained after this round of training. Among them, W is a positive integer.

[0135] An episode of training refers to the process in which the interaction between the agent and the data environment traverses all groups of training data in the training subset, and the deep Q-network is trained using K groups of experience data in the experience pool.

[0136] To facilitate understanding of the process of obtaining local model parameters in this embodiment, refer to Figure 3 the interaction process between the agent and the data environment shown in Figure 3 As can be seen, in each process, when the agent corresponding to the process obtains the state s from the data environment, it decides the action a. Then, the data environment returns the next state s' and the reward value r according to the action a. The experience pool (or replay memory unit) stores each piece of experience data (or interaction data) in the replay memory unit B, and then randomly extracts K groups of experience data and inputs them into the deep Q-network and the target network. The random gradient descent method is used to train the weight and bias parameters of the deep Q-network to maximize the Q function value. Every time W groups of experience data are traversed, the network parameters of the deep Q-network are synchronized to the target network until all groups of experience data are traversed, and the local model parameters of the deep Q-network are obtained.

[0137] Step 104: Based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training. The global model parameters obtained in this round of training are used to assign values to the model parameters of each reinforcement learning model before the next round of training.

[0138] In this embodiment, at the beginning of each round of training, the initialized local model parameters of each reinforcement learning model are the global model parameters. For the first round of training, the initialized global model parameters are the preset initial values. For the training rounds after the first round, the global model parameters are the parameter values determined based on the local model parameters of each reinforcement learning model after this round of training.

[0139] In one example, based on the local model parameters corresponding to each reinforcement learning model, determining the global model parameters obtained in this round of training includes: obtaining the local model parameters corresponding to each reinforcement learning model obtained in this round of training; determining the average value of each local model parameter to obtain the global model parameters obtained in this round of training. At this time, the global model parameters can be calculated by the following formula:

[0140]

[0141] where n represents the number of agents, which is the number of training subsets. represents the local model parameters corresponding to the kth reinforcement learning model obtained in this round of training, and W t+1 represents the global model parameters corresponding to this round of training. These global model parameters are used to assign values to the model parameters of each reinforcement learning model at the beginning of the next round of training.

[0142] In other embodiments, the weighted average value of each local model parameter may also be determined to obtain the global model parameter corresponding to the current round of training. The present embodiment does not limit the manner of determining the global model parameter.

[0143] In one example, obtaining the local model parameters corresponding to each reinforcement learning model obtained in the current round of training includes: reading the local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary. The shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model.

[0144] For example, in single-machine multi-process, the multiprocessing library is used for inter-process communication. Among them, shared memory is the key data transfer mechanism. Specifically, through the share_memory() method, the global model parameters are shared among each process. At the same time, the shared dictionary created by mp.Manager().dict() stores the network parameters obtained by each process during training. In this way, each process can store the trained local model parameters into this shared dictionary, realizing data transfer between processes.

[0145] Among them, the multiprocessing library is one of the standard libraries of Python, allowing developers to utilize multi-processes to execute parallel tasks. The multiprocessing library provides the share_memory() method, which allows data exchange between processes through shared memory. mp.Manager().dict() is another method of sharing resources in the multiprocessing library. It creates a server object in a child process, and other processes communicate with this server through the network to access and modify the shared data. In this way, the local model parameters can be accessed by accessing the shared dictionary created by mp.Manager().dict(), so as to calculate the global model parameters from the local model parameters and store them in the shared dictionary created by the share_memory() method.

[0146] Optionally, if the reinforcement learning models are trained synchronously among each process, the Barrier mechanism may also be used to ensure that each process synchronously performs model training and parameter update operations at an appropriate time.

[0147] In another example, obtaining the local model parameters corresponding to each reinforcement learning model obtained in the current round of training includes: calling a pre-created remote function to obtain the local model parameters obtained by each process for training the reinforcement learning model; where different processes are used to train different reinforcement learning models.

[0148] For example: Call a remote function and collect the results through the ray.get() function. Optionally, in the single-machine mode, the process information, task execution status, etc. can also be viewed through the Dashboard.

[0149] To more clearly understand the process of determining the global model parameters obtained in this round of training by averaging the local model parameters corresponding to each reinforcement learning model, an exemplary description of the update process of the global model parameters is given by taking the average of the local model parameters corresponding to each reinforcement learning model to obtain the global model parameters as an example. Refer to Figure 4 , the input Input of the parameter averaging algorithm for the global model parameters is the initialized global model parameter W 0 , the working node n (i.e., the process in the above text), and the threshold T of the number of training episodes; the output Output of the parameter averaging algorithm is the finally used global model parameter W T . For each round of training t ∈ [0, T - 1]; the master process Master broadcasts the current global model parameter W to each working node t , and each working node k ∈ [1, n] obtains the global model parameter W t . After that, as the initial model parameter of the reinforcement learning model corresponding to this working node, the reinforcement learning model is trained by the gradient descent method to obtain the local model parameters corresponding to this round of training The master process Master obtains the local model parameters obtained by each working node and updates the global model parameters according to the following formula.

[0150]

[0151] In the case where the number of training episodes t = T, the parameter averaging algorithm outputs W T .

[0152] Step 105, when the number of training episodes meets the preset conditions, determine the target reinforcement learning model based on the global model parameters obtained from the last round of training, so as to classify the input signal to be classified based on the target reinforcement learning model to obtain the classification result.

[0153] Determining the target reinforcement learning model based on the global model parameters obtained from the last round of training includes: using the network structure of the deep Q network in the agent and the global model parameters obtained from the last round of training to determine the target reinforcement learning model.

[0154] In one example, the number of training episodes meeting the preset conditions includes but is not limited to: the number of training episodes is equal to the preset threshold T of the number of training episodes; and / or, the change amount of the loss function value after the end of consecutive x rounds of training is less than the preset change amount threshold. The implementation manner of the preset conditions is not limited in this embodiment.

[0155] Optionally, after the target reinforcement learning model is obtained at the end of model training, the test data in the test set can also be used to test the target reinforcement learning model to obtain the classification result corresponding to the test data, so as to test the performance of the target reinforcement learning model.

[0156] In summary, the object classification method based on distributed multi-agent reinforcement learning provided in this embodiment divides the training set into at least two training subsets; wherein each training subset includes each first training data and part of the second training data; for each training subset, the corresponding agent of the training subset performs reinforcement learning on the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training, and the model parameters of different reinforcement learning models before training are all global model parameters; based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training; when the number of training rounds meets the preset conditions, determine the target reinforcement learning model based on the global model parameters obtained in the last round of training, so as to perform object classification on the input signal to be classified based on the target reinforcement learning model to obtain a classification result; it can solve the problem that the efficiency of traditional object classification methods is relatively low. Since the target reinforcement learning model is obtained through distributed training based on parameter averaging, when performing object classification, only one target reinforcement learning model is needed to achieve object classification, without the need to process through the reinforcement learning models corresponding to multiple agents respectively, which can ensure the object classification efficiency. At the same time, it can solve the problem that the classification ability of traditional object classification methods is limited in data imbalance and multi-modal scenarios. By reusing the first training data with a small quantity and having multiple agents perform division-of-labor learning on the second training data, the ability of the algorithm to process large-scale imbalanced data can be improved.

[0157] In addition, with the breakthrough of AlphaGo, deep reinforcement learning has become a recognized effective technology for solving continuous decision-making problems. Among them, AlphaGo is an intelligent robot that has defeated many professional Go players in human-machine Go games. At present, a large number of algorithms have been proposed to solve challenging problems between deep reinforcement learning (DRL) and real-world applications, such as exploration and exploitation dilemmas, data inefficiency, multi-agent cooperation and competition, etc. Among all these challenges, due to the trial-and-error learning mechanism of DRL requiring a large amount of interaction data, data inefficiency is the most criticized problem. To address this issue, inspired by distributed machine learning techniques, distributed deep reinforcement learning (DDRL) has been proposed and successfully applied in the fields of computer vision and natural language processing. In recent years, to better train large-scale deep reinforcement learning algorithms, dedicated training frameworks have been proposed, which support more complex real-time data sampling, model training, and inference requirements by abstracting algorithm interfaces and system designs. Existing typical distributed reinforcement learning frameworks include Acme, Sample-Factory, and SEEDRL, but these frameworks ignore the heterogeneity between multi-agent algorithms. For learning paradigms that emphasize agent interaction, such as centralized training and collaborative algorithms based on communication functions, no explicit support is provided, and there is a lack of corresponding unified interfaces to simplify algorithm implementation and training processes. In this embodiment, on the one hand, by providing a unified input-output interface for different types of multi-agent reinforcement learning algorithms, including standardized representations of information such as states, actions, and rewards, and unified setting methods for model structures and algorithm configuration parameters in different agents, algorithm interface standardization can be achieved; on the other hand, by setting an inter-agent communication protocol, communication between different processes and sharing of model parameters can be realized; on the other hand, the same processes can be set for data preprocessing, model training, model evaluation, etc. for development and testing, which can simplify the development and testing difficulties of each link.

[0158] In addition, regarding the development of the multi-agent reinforcement learning algorithm framework, existing work has mainly focused on algorithm implementation and has not paid much attention to the scalability of the algorithm in large-scale scenarios. Or more often, it is an algorithm library designed specifically for certain scenarios, such as the PyMARL and SMARTS frameworks. Therefore, for the scenarios supported by the multi-agent reinforcement learning algorithm framework, there has always been a lack of a comprehensive framework to implement the processes of algorithm implementation, training, and deployment testing. In this embodiment, the framework of the object classification method can support algorithm training and testing in large-scale scenarios, including processing large-scale data sets, parallel training, etc. At the same time, the algorithm can further expand the data set to achieve support for large-scale scenario applications. At the same time, the framework of the object classification method can easily integrate new multi-agent reinforcement learning algorithms and support flexible configuration and modification of the algorithms, such as adding or reducing data modalities, adding or reducing agents for configuration, etc., to improve the scalability of the algorithm.

[0159] To more clearly understand the object classification method based on distributed multi-agent reinforcement learning proposed in this application, the following takes the application of this object classification method to a fault diagnosis scenario and the second training data including 3 modalities as an example for illustration. At this time, the target reinforcement learning model is used to diagnose faults in the device; correspondingly, the signal to be classified includes the device signal of the device; the first object classification includes fault signal classification, and the first classification label includes at least one fault classification label for fault signal classification; the second object classification includes normal signal classification, and the second classification label includes normal classification labels. Refer to Figure 5 This method includes the following steps:

[0160] Step 51, collection and division of the data set.

[0161] Suppose the CWRU data set provided by Case Western Reserve University (CWRU) is used as the original data set. In the CWRU data set, there are differences in working conditions for the same type of device, resulting in differences in the data monitored for normal classification, that is, there are different modalities. Since the data for normal classification is huge, therefore, the agents can only exchange the data for fault classification and cannot exchange the data for normal classification. Taking the data for normal classification at different horsepowers as an example to simulate the data for normal classification with different modalities, the data for fault classification that can be interacted between multiple devices is introduced. A total of 30,000 groups of data are selected for the experiment, and the imbalance ratio of the overall data set (that is, the ratio of the number of data for normal classification to the number of data for fault classification) is 21:1. The specific data set description is shown in Table 1 above. The 30,000 samples are divided into a training set and a test set according to 8:2, where the training set has 24,000 groups of training data, as shown in Table 2 above, and the test set has 6,000 groups of test data.

[0162] Step 52: Create a process for each modality and allocate a training subset to each process.

[0163] According to the number of modalities of the second training data in the training set, 3 agents are selected to interact with the data environment. For the 24,000 samples in the training set, specific training subsets S1, S2, and S3 are allocated to the 3 agents respectively through the reuse of the first training data (i.e., the training data for fault classification) and the division of labor learning for different modalities. The specific situation of the sample subset corresponding to each agent is shown in Table 3 above.

[0164] Step 53: Construct a reinforcement learning model.

[0165] Map each classification label in the training set to an action set A = {0, 1, 2, 3}. Among them, 0 represents normal classification, 1 represents the IR007 fault type in fault classification, 2 represents the B007 fault type in fault classification, and 3 represents the OR007 fault type in fault classification. The state space of the agent corresponding to each process is the sample data in the training subset corresponding to that process.

[0166] For the agent corresponding to each process, the state s of the agent at each time step t will correspond to different sample data x in the training subset t . At the beginning of training, the agent will randomly receive the first sample data x1 returned by the data environment as its initial state s1, and determine the corresponding classification action based on this initial state s1. After traversing all the sample data in the training subset, one episode of training ends. When a new episode starts, the data environment will shuffle the order of the sample data in the training subset and interact with the agent again in a new order.

[0167] In each iteration, the agent corresponding to each process makes an action according to the data state returned by the data environment in combination with the ε-greedy policy. When the node corresponding to the actually determined action is consistent with the classification label corresponding to the sample data, the reward value is set to +1, otherwise the reward value is -1. The relevant descriptions of the ε-greedy policy and the reward function are shown in the above embodiments, and will not be elaborated in this embodiment.

[0168] Step 54: Initialize the parameters and perform parallel training.

[0169] The reinforcement learning models corresponding to different processes are initialized with global model parameters and distributed to each process. Each process uses the experience data in its local experience pool for parallel training, updates the local model parameters through the reinforcement learning algorithm, and stores them in the shared dictionary.

[0170] Step 55: Exchange and update the parameters.

[0171] After an episode ends, each process completes the training of the local reinforcement learning model. The local model parameters corresponding to each process obtained from the shared dictionary are averaged, which is used as the new global parameter model, and the updated global parameters are broadcast to each process. After a certain number of rounds of training, the trained deep Q-network parameters are obtained, and the target reinforcement learning model is determined based on these deep Q-network parameters.

[0172] Step 56: Model testing.

[0173] The test data in the test set is input into the target reinforcement learning model to obtain the classification result for model testing.

[0174] It has been verified that the object classification method based on distributed multi-agent reinforcement learning provided by this application has higher accuracy and classification performance. Specifically, still taking the use of CWRU as the original data set as an example, this data set includes data for normal classification and data for fault classification. The experimental vibration data is the acceleration data collected from the drive end (DE), with a sampling frequency of 12 kHz. The bearing test bench device used to obtain the CWRU data set is as Figure 6 shown. The data set consists of nine types of fault classification data and one type of normal classification data. The fault types of the fault classification include outer race fault (OR), inner race fault (IR), and roller fault (B), and the degree of the fault in the fault also has different diameters, including 0.007 mm, 0.014 mm, and 0.021 mm. The detailed information of the fault classification and the detailed information of the normal classification refer to Table IV below.

[0175] Table IV:

[0176] Label Fault Diameter (in.) Fault Type Alias 0 0 Normal Normal_097 1 0.007 IR IR007_105 2 0.007 B B007_118 3 0.007 OR OR007_130 4 0.014 IR IR014_169 5 0.014 B B014_185 6 0.014 OR OR014_197 7 0.021 IR IR021_209 8 0.021 B B021_222 9 0.021 OR OR021_234

[0177] In a possible implementation manner, in the scenario of solving the fault diagnosis problem under data imbalance, accuracy cannot fully reflect the performance of the model, and the F value is more suitable for reflecting the performance of the model in an imbalanced data set. Therefore, this embodiment takes the introduction of the F value as an evaluation index for illustration. Before explaining the F value, the meanings of precision and recall are first introduced. Precision represents the proportion of positive example samples correctly predicted by the classifier (i.e., the target reinforcement learning model in this application) among all predicted positive example samples, while recall represents the proportion of all actual positive example samples correctly predicted as positive by the classifier. Precision focuses on the accuracy of the prediction result, and recall focuses on the comprehensive coverage of positive examples. However, there is usually a trade-off relationship between these two indicators, so the F value is introduced as a comprehensive evaluation index. Denote the F value of classification c as F c , F c The calculation formula of can be expressed by the following formula:

[0178]

[0179] Among them, Precision represents the precision rate of class c, and Recall represents the recall rate of class c; the higher the F value, the better the performance of the classifier. In the multi-classification problem, the formula for calculating the final F value can be expressed as follows:

[0180]

[0181] Among them, C represents the total number of classifications, and F c represents the F value of class c.

[0182] Suppose that the network structures of the deep Q-network and the target network in each agent in the above embodiments both include: two linear layers, which are set to 128 neurons and 32 neurons respectively. The number of neurons in the input layer is the dimension of the sample, that is, 1200, and the number of neurons in the output layer is 4, which represent the normal class and 3 fault classes respectively, for a total of four categories. The division method of the training subset can refer to Table 3 above. The parameter settings in deep reinforcement learning are as follows: ε = 0.1 in the ε-greedy policy, the weight parameter δ = 0.001 in the loss function, the number of experience data randomly selected from the experience pool K = 64, and the threshold of the training round episode = 20.

[0183] Taking the single-agent reinforcement learning method and the group decision-making method based on multi-agent reinforcement learning as an example for comparative experiments with the object classification provided in this embodiment. Among them, the single-agent reinforcement learning method means using 24,000 training samples for reinforcement learning training, and the data imbalance ratio is 21:1; the idea of the group decision-making method based on multi-agent reinforcement learning for allocating the data set is consistent with this article, which is to allocate specific training set samples for each agent through the idea of reusing fault samples and dividing labor learning for multi-modal. Each agent has 10,000 training samples, and the imbalance ratio is 7:1. However, in the training stage of the group decision-making method, each agent perceives the data features through the neural network and performs policy learning according to the reward value feedback by the data environment, so as to obtain the optimal policy of each agent. In the final test stage, the samples are input into the models corresponding to each agent, and each agent generates its own classification result, and the final classification result is obtained through the group decision-making methods of voting and fuzzy integral. The specific experimental results are shown in Table 5 below.

[0184] Table 5:

[0185]

[0186] As can be seen from Table V, the method based on multi-agent reinforcement learning has higher accuracy and F-value compared to the single-agent reinforcement learning method. This indicates that the multi-agent system effectively alleviates the problems of data imbalance and multi-modal within the normal class through techniques such as fault sample reuse and division of labor learning for multi-modal. Comparing with the multi-agent reinforcement learning method, it is found that the distributed parameter averaging training method adopted in this application is superior to the group decision-making method, with the accuracy increased by 2.25% and the F-value increased by 0.0957. This shows that the way of enabling multiple agents to interact information during the training stage through the parameter averaging method in this application helps to improve the diagnostic effect of the algorithm.

[0187] In addition, Table VI shows the training time of the single-agent reinforcement learning and distributed reinforcement learning methods.

[0188] Table VI:

[0189]

[0190] Theoretically analyzed, the distributed reinforcement learning method uses 10,000 samples for training in each process by reusing the fault class samples, while the single-agent reinforcement learning method uses 24,000 samples for training. Therefore, the efficiency of the distributed reinforcement learning method should be 24,000 / 10,000 = 2.4 times that of the single-agent reinforcement learning method. However, according to the results in Table VI, the efficiency of the distributed reinforcement learning method is 1.79 times that of the single-agent reinforcement learning method. This is because after each episode ends, the processes need to perform parameter interaction, which will sacrifice some time, in line with the actual situation. The experimental results in Table VI further verify that the distributed training method based on parameter averaging in this application improves the efficiency of the algorithm.

[0191] In summary, the object classification method proposed in this application can solve the problems of data imbalance and multi-modal in a certain type of data in the object classification scenario. It uses multi-agents to perform fault diagnosis on the training set, and establishes an object classification method based on distributed multi-agent reinforcement learning through techniques such as fault sample reuse, division of labor learning for multi-modal, and distributed training based on parameter averaging. It successfully overcomes the limitations of traditional single-agent algorithms in dealing with data imbalance and multi-modal within the normal category, and significantly improves the ability of the algorithm to process imbalanced data and the efficiency of processing large-scale imbalanced data.

[0192] Figure 7 It is a block diagram of an object classification device based on distributed multi-agent reinforcement learning shown according to an exemplary embodiment. The device at least includes the following modules: a data acquisition module 710, a data division module 720, a reinforcement learning module 730, a parameter update module 740, and an object classification module 750.

[0193] A data acquisition module 710, configured to acquire a training set, where the training set includes multiple groups of first training data and multiple groups of second training data; wherein, the number of the first training data is less than the number of the second training data; each group of first training data includes a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of second training data includes a second sample signal corresponding to a second object classification and a second classification label of the second object classification;

[0194] A data division module 720, configured to divide the training set to obtain at least two training subsets; wherein, each training subset includes each piece of first training data and a part of the second training data, and the second training data in different training subsets is different;

[0195] A reinforcement learning module 730, configured to, for each training subset, perform reinforcement learning on the training subset through an agent corresponding to the training subset to obtain local model parameters obtained by the reinforcement learning model of the agent after this round of training; the model parameters of different reinforcement learning models before training are all global model parameters, and the agents corresponding to different training subsets are different;

[0196] A parameter update module 740, configured to determine global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model, and the global model parameters obtained in this round of training are used to assign values to the model parameters of each reinforcement learning model before the next round of training;

[0197] An object classification module 750, configured to, when the number of training rounds meets a preset condition, determine a target reinforcement learning model based on the global model parameters obtained in the last round of training, so as to perform object classification on an input signal to be classified based on the target reinforcement learning model to obtain a classification result.

[0198] For related descriptions, see the above method embodiments.

[0199] In some embodiments, the functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the descriptions of the above method embodiments. For the sake of brevity, details are not described here again.

[0200] The embodiments of the present disclosure also propose a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above methods are implemented. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0201] The embodiments of the present disclosure also propose an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein, the processor is configured to implement the above methods when executing the instructions stored in the memory.

[0202] The embodiments of the present disclosure also provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in the processor of an electronic device, the processor in the electronic device executes the above method.

[0203] Figure 8 It is a block diagram of an object classification device 1900 based on distributed multi-agent reinforcement learning shown according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Referring to Figure 8 , the device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0204] The device 1900 may further include a power component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0205] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions. The above computer program instructions can be executed by the processing component 1922 of the device 1900 to complete the above method.

[0206] The embodiments of the present disclosure have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the disclosed embodiments.

Claims

1. An object classification method based on distributed multi-agent reinforcement learning, characterized in that, The method includes: Obtaining a training set, where the training set includes multiple groups of first training data and multiple groups of second training data; wherein, the number of the first training data is less than the number of the second training data; each group of first training data includes a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of second training data includes a second sample signal corresponding to a second object classification and a second classification label of the second object classification; Dividing the training set to obtain at least two training subsets; wherein, each training subset includes each piece of first training data and some of the second training data, and the second training data in different training subsets is different; For each training subset, performing reinforcement learning on the training subset through an agent corresponding to the training subset to obtain local model parameters obtained by the reinforcement learning model of the agent after this round of training; the model parameters of different reinforcement learning models before training are all global model parameters, and the agents corresponding to different training subsets are different; Obtaining the local model parameters corresponding to each reinforcement learning model obtained in this round of training; determining the average value of each local model parameter to obtain the global model parameters obtained in this round of training, and the global model parameters obtained in this round of training are used to assign values to the model parameters of each reinforcement learning model before the next round of training; When the number of training rounds meets a preset condition, determining a target reinforcement learning model based on the global model parameters obtained in the last round of training, so as to perform object classification on an input signal to be classified through the target reinforcement learning model to obtain a classification result; the target reinforcement learning model is used for fault diagnosis of a device; correspondingly, the signal to be classified includes a device signal of the device; the first object classification includes a fault signal classification, and the first classification label includes at least one fault classification label of the fault signal classification; the second object classification includes a normal signal classification, and the second classification label includes a normal classification label.

2. The method according to claim 1, wherein The second training data is multi-modal data; correspondingly, the dividing the training set to obtain at least two training subsets includes: Dividing the second training data according to the modality of the second training data to obtain the second training data corresponding to each modality; Combining the second training data corresponding to each modality with each group of first training data to obtain a training subset corresponding to each modality.

3. The method according to claim 2, wherein The multi-modal data includes second sample signals generated by a generation object of the second sample signal under different working conditions, and different working conditions correspond to different modalities.

4. The method according to claim 1, wherein The for each training subset, performing reinforcement learning on the training subset through an agent corresponding to the training subset to obtain local model parameters obtained by the reinforcement learning model of the agent after this round of training includes: Creating multiple processes, where the processes correspond to the agents one by one; Performing reinforcement learning on the training subset through the multiple processes running the corresponding agents in parallel to obtain the local model parameters obtained after this round of training.

5. The method according to claim 1, characterized in that, For each training subset, the training subset is subjected to reinforcement learning by the agent corresponding to the training subset to obtain the local model parameters obtained by the reinforcement learning model of the agent after this round of training, including: For each training subset, a training order of each group of training data in this round of training of the training subset is generated to obtain a data environment; wherein, the training orders corresponding to different rounds of training are different; the training data includes first training data and second training data in the training subset; The i-th sample data in the i-th group of training data is sequentially returned to the agent corresponding to the training subset by the data environment according to the training order; the agent includes the reinforcement learning model, the reinforcement learning model includes a deep Q-network and a target network, and the network parameters of the deep Q-network and the target network are initialized to be the same; i is a positive integer sequentially taking values from 1 to n, and n is the total number of training data in the training subset; The i-th classification action corresponding to the i-th sample data is determined by the deep Q-network; wherein, the i-th classification action indicates one of the first classification label and the second classification label; The i-th classification action is returned by the agent to the data environment for the data environment to determine the i-th reward of the i-th classification action according to the i-th classification action and the classification label corresponding to the sample data, and return the i-th reward and the (i + 1)-th sample data indicated by the training order to the agent; The i-th experience data is stored in the experience pool of the agent, and i = i + 1 is updated to trigger the execution of the step of sequentially returning the i-th sample data in the i-th group of training data to the agent corresponding to the training subset by the data environment and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward, and the (i + 1)-th sample data; After all groups of training data in the training subset are traversed, K groups of experience data are randomly selected from the experience pool; K is a positive integer; Each group of experience data is respectively input into the deep Q-network and the target network to update the deep Q-network based on the difference between the first network result of the deep Q-network and the second network result of the target network; Every time W groups of experience data are traversed, the network parameters of the deep Q-network are synchronized to the target network until all groups of experience data are traversed to obtain the network parameters of the deep Q-network obtained in this round of training; the network parameters of the deep Q-network obtained in this round of training are the local model parameters obtained after this round of training, and W is a positive integer.

6. The method according to claim 1, wherein The obtaining of the local model parameters corresponding to each reinforcement learning model obtained in this round of training includes: Reading the local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary, wherein the shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model; Or, Call the pre-created remote function to obtain the local model parameters obtained by each process training the reinforcement learning model; wherein, different processes are used to train different reinforcement learning models.

7. An object classification device based on distributed multi-agent reinforcement learning, characterized in that, It includes: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1 to 6 when executing the instructions stored in the memory.

8. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images