Object classification method and device based on distributed multi-agent reinforcement learning

Abstract

The invention relates to the technical field of computers, and discloses an object classification method and device based on distributed multi-agent reinforcement learning. Dividing the training set to obtain at least two training subsets; for each training subset, reinforcement learning is carried out on the training subset through the intelligent agent corresponding to the training subset, local model parameters obtained after the training of the reinforcement learning model of the intelligent agent are obtained, and model parameters of different reinforcement learning models before training are global model parameters; on the basis of the local model parameters corresponding to the reinforcement learning models, determining global model parameters obtained by the round of training; under the condition that the training round meets a preset condition, determining a target reinforcement learning model based on the global model parameters obtained by the last round of training, and performing object classification on the input to-be-classified signal based on the target reinforcement learning model to obtain a classification result; the object classification efficiency can be ensured, and the capability of processing large-scale unbalanced data by an algorithm is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an object classification method and device based on distributed multi-agent reinforcement learning. Background Art

[0002] With the development of artificial intelligence, it is widely used to classify objects by training neural network models to obtain object classification models. For example, equipment fault classification is performed through object classification models, text classification is performed through object classification models, etc. In the traditional object classification model training method, it is necessary to use training data of different categories to train the neural network model. However, in some scenarios, the training data of different categories are usually unbalanced, that is, some categories have more training data and other categories have less training data. In this case, the classification performance of the trained object classification model is low. For example, in the scenario of equipment fault classification, the number of training data of general fault categories is small, while the number of training data of normal categories is large.

[0003] In order to improve the performance of model classification when the training data is unbalanced, an object classification method based on agent reinforcement learning is proposed. The traditional object classification method based on agent reinforcement learning includes: obtaining an unbalanced training set; using the training set to build a data environment; based on the Deep Q-Leaning Network (DQN) algorithm, the reinforcement learning model is trained through the interaction between the data environment and the agent to obtain the object classification model.

[0004] However, although the object classification method based on agent reinforcement learning can overcome the problem of imbalanced training set data, the object classification model obtained through single agent training has limited classification ability and may not achieve the expected classification performance. Summary of the invention

[0005] In view of this, the present disclosure proposes an object classification method and device based on distributed multi-agent reinforcement learning, which can achieve more effective recognition under unbalanced data, wherein each agent is responsible for processing a specific training subset, and the classification result of each sample is determined by its corresponding agent, which can improve the ability of the object classification method to handle unbalanced data and the efficiency of processing large-scale unbalanced data.

[0006] According to one aspect of the present disclosure, a method for object classification based on distributed multi-agent reinforcement learning is provided, the method comprising:

[0007] Acquire a training set, the training set comprising a plurality of groups of first training data and a plurality of groups of second training data; wherein the number of the first training data is less than the number of the second training data; each group of the first training data comprises a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of the second training data comprises a second sample signal corresponding to a second object classification and a second classification label of the second object classification;

[0008] Dividing the training set into at least two training subsets; wherein each training subset includes each first training data and a portion of the second training data, and the second training data in different training subsets are different;

[0009] For each training subset, the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain local model parameters of the agent's reinforcement learning model after this round of training; the model parameters of different reinforcement learning models before training are all global model parameters, and different training subsets correspond to different agents;

[0010] Based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training, wherein the global model parameters obtained in this round of training are used to assign values ​​to the model parameters of each reinforcement learning model before the next round of training;

[0011] When the training rounds meet the preset conditions, the target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to perform object classification on the input signal to be classified based on the target reinforcement learning model to obtain a classification result.

[0012] In a possible implementation, the second training data is multimodal data; accordingly, the dividing the training set to obtain at least two training subsets includes:

[0013] Dividing the second training data according to the modality of the second training data to obtain second training data corresponding to each modality;

[0014] The second training data corresponding to each modality is merged with each group of first training data to obtain a training subset corresponding to each modality.

[0015] In a possible implementation manner, the multimodal data includes second sample signals generated by a generating object of the second sample signal under different working conditions, and different working conditions correspond to different modes.

[0016] In a possible implementation, for each training subset, performing reinforcement learning on the training subset through an agent corresponding to the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training includes:

[0017] Creating a plurality of processes, wherein the processes correspond one to one with the intelligent agents;

[0018] The corresponding intelligent agents are run in parallel by the multiple processes to perform reinforcement learning on the training subset to obtain local model parameters after this round of training.

[0019] In a possible implementation, for each training subset, performing reinforcement learning on the training subset through an agent corresponding to the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training includes:

[0020] For each training subset, a training sequence of each group of training data in the training subset in the current round of training is generated to obtain a data environment; wherein different training sequences correspond to different rounds of training; and the training data include first training data and second training data in the training subset;

[0021] The data environment returns the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence according to the training order; the agent includes the reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer taking values ​​from 1 to n in sequence, and n is the total number of training data in the training subset;

[0022] Determining, by means of the deep Q network, an i-th classification action corresponding to the i-th sample data; wherein the i-th classification action indicates one of the first classification label and the second classification label;

[0023] Returning the i-th classification action to the data environment through the agent, so that the data environment determines the i-th reward of the i-th classification action according to the classification label corresponding to the i-th classification action and the sample data, and returns the i-th reward and the i+1th sample data indicated by the training sequence to the agent;

[0024] The i-th experience data is stored in the experience pool of the agent, and i=i+1 is updated to trigger the step of returning the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence through the data environment according to the training order and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data;

[0025] After each set of training data in the training subset is traversed, K sets of experience data are randomly selected from the experience pool; K is a positive integer;

[0026] Inputting each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network;

[0027] Every time W groups of experience data are traversed, the network parameters of the deep Q network are synchronized to the target network until the traversal of each group of experience data is completed, and the network parameters of the deep Q network obtained in this round of training are obtained; the network parameters of the deep Q network obtained in this round of training are the local model parameters obtained after the current round of training, and W is a positive integer.

[0028] In a possible implementation, determining the global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model includes:

[0029] Get the local model parameters corresponding to each reinforcement learning model obtained in this round of training;

[0030] Determine the average value of each local model parameter to obtain the global model parameter obtained in this round of training.

[0031] In a possible implementation, obtaining local model parameters corresponding to each reinforcement learning model obtained in this round of training includes:

[0032] Reading local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary, wherein the shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model;

[0033] or,

[0034] Call a pre-created remote function to obtain local model parameters obtained by each process training the reinforcement learning model; different processes are used to train different reinforcement learning models.

[0035] In one possible implementation, the target reinforcement learning model is used to perform fault diagnosis on the device; accordingly, the signal to be classified includes the device signal of the device; the first object classification includes fault signal classification, and the first classification label includes at least one fault classification label of the fault signal classification; the second object classification includes normal signal classification, and the second classification label includes a normal classification label.

[0036] According to another aspect of the present disclosure, there is provided an object classification device based on distributed multi-agent reinforcement learning, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0037] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0038] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0039] At least two training subsets are obtained by dividing the training set; wherein each training subset includes each first training data and part of the second training data; for each training subset, the training subset is reinforced learned by the intelligent agent corresponding to the training subset to obtain the local model parameters of the intelligent agent's reinforcement learning model after this round of training, and the model parameters of different reinforcement learning models before training are all global model parameters; based on the local model parameters corresponding to each reinforcement learning model, the global model parameters obtained in this round of training are determined; when the training rounds meet the preset conditions, the target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to classify the input signal to be classified based on the target reinforcement learning model to obtain the classification result; the problem of low efficiency of traditional object classification methods can be solved. Since the target reinforcement learning model is obtained by distributed training based on parameter averaging, when classifying objects, only one target reinforcement learning model is used to realize object classification, and there is no need to process through the reinforcement learning models corresponding to multiple intelligent agents, so that the efficiency of object classification can be guaranteed. At the same time, it can solve the problem of limited classification ability of traditional object classification methods in data imbalance and multimodal scenarios. By reusing a small amount of first training data and having multiple agents divide the work of learning the second training data, the algorithm's ability to handle large-scale imbalanced data can be improved.

[0040] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0042] Figure 1 A flowchart showing an object classification method based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure is shown;

[0043] Figure 2 A schematic diagram showing the state movement of an agent at each time step in a round of training according to an embodiment of the present disclosure;

[0044] Figure 3 A schematic diagram showing a deep learning process according to an embodiment of the present disclosure;

[0045] Figure 4 A schematic diagram showing a code implementation of a parameter averaging algorithm according to an embodiment of the present disclosure;

[0046] Figure 5 A schematic diagram showing a fault diagnosis method based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure is shown;

[0047] Figure 6 A schematic diagram showing a bearing test bench device according to an embodiment of the present disclosure is shown;

[0048] Figure 7 A block diagram of an object classification device based on distributed multi-agent reinforcement learning according to an embodiment of the present disclosure is shown;

[0049] Figure 8 A block diagram of an object classification device based on distributed multi-agent reinforcement learning according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0051] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0052] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0053] First, several terms involved in this application are introduced.

[0054] Reinforcement Learning (RL) is a machine learning method that achieves decision-making and behavior optimization through interactive learning between an agent and the environment. In reinforcement learning, the agent explores the environment by trying different actions and adjusts its strategy based on the feedback (rewards) from the environment to maximize the long-term accumulated reward value. Among them, the agent and the environment are the two core components of reinforcement learning.

[0055] Agent: It is the decision maker in the reinforcement learning system, which achieves a certain goal by perceiving the environment and performing actions. The agent can be a robot, a software program, or any entity that can interact with the environment and take actions.

[0056] Environment: It is the external world in which the agent is located, including everything that the agent can perceive and influence. The environment provides the agent with status information and gives corresponding feedback based on the agent's actions. In this application, the agent is a software program, the environment is built based on training data, and the action indicates one of the multiple classification labels in the training data. At this time, the agent interacts with the environment by deciding the classification label, and the environment provides reward feedback to the agent based on the classification label.

[0057] Interaction relationship: The interaction between the agent and the environment is carried out through a cyclic process. The agent perceives the current state (s) at each time step, such as: the current state is the sample signal in the training data fed back by the environment, and selects an action (a) based on the adopted strategy (π), such as: the action indicates a classification decided by the agent based on the sample signal; then the action is executed, and the environment updates the state and gives the agent a reward (r), such as updating the current state to the sample signal in the next set of training data, and the reward for the classification (or action) corresponding to the current state. This process is repeated continuously, allowing the agent to learn the optimal strategy through trial and error to maximize the cumulative reward.

[0058] In reinforcement learning, state, action, and reward are the basic elements to describe the interaction between an agent and its environment.

[0059] State (State, s): The state is a description of the environment at a certain moment. It can be a comprehensive description of the environment or a collection of certain key features. The state provides the agent with the information needed for decision-making. The representation of the state can be discrete (such as the position on the chessboard) or continuous (such as the position information of the robot). In this application, the state is the sample signal in each set of training data.

[0060] Action (a): An action is an operation that an agent can perform in a given state. The action space can be a finite set of discrete actions or an infinite set of continuous actions. The agent affects the environment by selecting actions and transitions to a new state. In this application, an action indicates one of the classification labels corresponding to each sample signal.

[0061] Reward (r): Reward is the feedback signal from the environment to the action performed by the agent, which is used to measure the goodness of the action in the current state. The reward function defines the immediate return for each state-action pair. The goal of the agent is to maximize its cumulative reward by choosing appropriate actions.

[0062] The goal of reinforcement learning is to find a strategy that maximizes the cumulative reward of the agent in its interactions with the environment.

[0063] Policy (π): It is the rule or function for the agent to choose an action in each state. It can be deterministic (choose a fixed action in each state) or stochastic (choose different actions with a certain probability in each state). The policy can be expressed as π(s) = a, indicating that action a is chosen in state s, or π(a|s), indicating the probability of choosing action a in state s.

[0064] Value Function (V): It is used to evaluate the expected cumulative reward of a state when executing a specific strategy. The state value function V(s) represents the expected cumulative reward that can be obtained by following the strategy π starting from state s.

[0065] Q-Function (Q): The Q-Function (state-action value function) evaluates the expected cumulative reward that can be obtained by performing a specific action in a specific state. The Q-value function Q(s, a) represents the expected cumulative reward that can be obtained by performing action a in state s and then following the policy π.

[0066] Value function and Q function are key tools in reinforcement learning, used to evaluate and optimize strategies. By learning and updating these functions, the agent can gradually improve its strategy and reach the optimal decision.

[0067] Reinforcement learning covers a variety of core algorithms, each of which is designed to solve different types of decision-making problems. These algorithms include value iteration, policy iteration, Q learning, Deep Q-Leaning Network (DQN), policy gradient methods, and actor-critic methods. They handle the interaction between the agent and the environment in different ways and maximize the expected cumulative reward by optimizing the strategy. Among them, DQN uses the experience replay mechanism to store the experience data of the agent and randomly samples small batches of data for training, which can break the data correlation and improve the model performance under unbalanced training data. Based on this, in this application, the DQN method for processing the interaction between the agent and the environment is used as an example for explanation.

[0068] Below, the object classification method based on distributed multi-agent reinforcement learning provided by the present application is described in detail. The application scenarios of the object classification method provided by the present application include but are not limited to: fault diagnosis scenarios (i.e., classifying whether the classified signal is a fault signal and the fault type), text classification scenarios (i.e., classifying which word the classified signal corresponds to), etc. This embodiment does not limit the application scenarios of the object classification method.

[0069] Taking the fault diagnosis scenario as an example, fault diagnosis is the core content of system operation and maintenance and a key step to ensure the normal operation of the system. Through effective fault diagnosis, the root cause of the problem can be quickly located and correct measures can be taken to repair it, thereby minimizing system downtime and improving production efficiency. Therefore, timely and accurate fault diagnosis is crucial to the normal operation of industrial systems.

[0070] Traditional fault diagnosis methods mainly rely on expert knowledge and rules, and need to formulate fault models and rule bases based on experience. However, these models can only describe the signal characteristics of a few well-defined fault types, but the actual natural faults are often more complex, and traditional methods cannot cope with large-scale and complex fault diagnosis needs. In order to overcome the limitations of traditional methods, data-driven fault diagnosis methods have received widespread attention in recent years. These methods use big data and machine learning technology to automatically learn fault characteristics and patterns through the analysis and mining of massive historical fault data, and perform fault diagnosis based on this. In the current rapidly developing information technology environment, the use of data-driven intelligent fault diagnosis methods has become an indispensable choice.

[0071] With the rapid development of artificial intelligence technology, intelligent fault diagnosis is realized by using artificial intelligence technology, making the fault diagnosis process more intelligent and automated. Recently, deep neural networks such as deep autoencoder (DAE) and deep convolutional neural network (DCNN) have been widely used to build end-to-end intelligent diagnosis models, reducing the dependence on manual experience and expert knowledge, and greatly promoting the development of intelligent fault diagnosis. However, although deep learning models have made certain progress in the field of industrial process fault diagnosis, they still face many difficulties and challenges. In actual engineering scenarios, machines usually work in normal conditions, which means that the fault data that can be collected is limited, resulting in the problem of unbalanced fault samples. Unbalanced fault samples refer to the uneven distribution of the number of samples of different categories in the data set, that is, the number of samples of the fault category is small, while the number of samples of the normal category is large. Intelligent fault diagnosis under unbalanced data refers to the use of limited machine fault samples to establish a fault diagnosis model to achieve accurate fault identification. Traditional fault diagnosis methods under unbalanced data include but are not limited to: strategies based on data enhancement, strategies based on feature learning, and strategies based on classifier design. Strategies based on data augmentation can improve the performance of diagnostic models by enhancing training data. Strategies based on feature learning can improve diagnostic performance by designing regularized neural networks or feature adaptation to extract features from limited fault data. Strategies based on classifier design can achieve high-precision fault diagnosis by building classifiers suitable for imbalanced data. In addition, in the case where a few samples are extremely scarce, some knowledge-driven methods with limited sample data, such as meta-learning and zero-shot learning, are proposed. At the same time, a general imbalanced sample selection strategy DiagSelect based on deep reinforcement learning is proposed to solve the data imbalance problem in the field of fault diagnosis through deep reinforcement learning. However, the fault classification ability of the fault diagnosis model obtained by training a single agent is limited and may not achieve the expected classification performance.

[0072] In one possible implementation, data processing efficiency and classification performance can be improved by introducing a multi-agent system. However, due to the huge number of normal samples, normal samples may not be shared between agents. Therefore, how to achieve accurate and efficient fault diagnosis in the case of data imbalance and multi-modality of normal samples becomes an important research issue.

[0073] In addition, after the introduction of the multi-agent system, reinforcement learning models corresponding to multiple agents will be obtained after training is completed. At this time, in the actual fault diagnosis process, each signal to be classified needs to be decided by the reinforcement learning models corresponding to multiple agents respectively, and the decision results must be integrated to obtain the final diagnosis result. At this time, it will lead to the problem of low diagnostic efficiency.

[0074] Object classification methods in other scenarios also have the above-mentioned problems of low classification efficiency, data imbalance, and limited classification ability in multimodal scenarios, which will not be elaborated in detail in this embodiment.

[0075] Based on the above technical problems, in the present application, multiple agents are used to perform reinforcement learning using at least two training subsets in the training set respectively, that is, each agent is responsible for processing a specific training subset, and the classification result of the sample signal in each group of training data is determined by its corresponding agent. An object classification method based on multi-distributed multi-agent reinforcement learning is established. This method can improve the ability of the algorithm to handle large-scale unbalanced data by reusing a small amount of first training data and dividing the learning of the second training data by multiple agents; at the same time, a target reinforcement learning model is obtained based on distributed training based on parameter averaging. When classifying objects, only one target reinforcement learning model is used to achieve object classification, without the need for processing by reinforcement learning models corresponding to multiple agents, thereby ensuring object classification efficiency.

[0076] Figure 1 The flowchart of the object classification method based on distributed multi-agent reinforcement learning according to one embodiment of the present disclosure is shown. In this embodiment, the method is described by using an electronic device with processing capabilities such as a user terminal or a server as an example. The user terminal includes but is not limited to: a computer, a tablet computer, a mobile phone, etc. This embodiment does not limit the implementation method of the electronic device. Figure 1 As shown, the method includes:

[0077] Step 101, obtaining a training set.

[0078] The training set includes multiple groups of training data, and the multiple groups of training data include multiple groups of first training data and multiple groups of second training data. The number of first training data is less than the number of second training data; that is, the training set is an unbalanced data set, and the number of first training data is small. Each group of training data includes a sample signal and a classification label corresponding to the sample signal. For example: each group of first training data includes a first sample signal corresponding to a first object classification and a first classification label for the first object classification; each group of second training data includes a second sample signal corresponding to a second object classification and a second classification label for the second object classification.

[0079] Taking the object classification scenario as a fault diagnosis scenario as an example, the first training data may be fault data, and accordingly, the first training data includes a fault signal and a fault classification label corresponding to the fault signal, and the fault classification label may be one or at least two. When the fault classification label includes at least two, different fault classification labels are used to represent different fault classifications. The second training data may be normal data.

[0080] The training set can be obtained by dividing the original data set in proportion. For example, the original data set is shown in Table 1 below. Each group of data in the original data set includes a sample signal and a classification label corresponding to the sample signal. The classification labels include fault classification labels 1, 2, 3, and normal classification label 0; fault classification label 1 indicates that the fault classification is IR007, and the number of sample data of fault classification label 1 is 1250; fault classification label 2 indicates that the fault classification is B007, and the number of sample data of fault classification label 2 is 1250; fault classification label 3 indicates that the fault classification is OR007, and the number of sample data of fault classification label 3 is 1250. The sample signal of normal classification label 0 includes three modes, namely: Normal-0hp (indicating that the working condition is 0 horsepower), Normal-1hp (indicating that the working condition is 1 horsepower) and Normal-2hp (indicating that the working condition is 2 horsepower), wherein the number of sample signals of each mode is 8750. Thus, the original data set includes 30,000 groups of data.

[0081] Table 1:

[0082] Label category Modal Sample size 0 Normal Normal-0hp 8750 0 Normal Normal-1hp 8750 0 Normal Normal-2hp 8750 1 IR007 None 1250 2 B007 None 1250 3 OR007 None 1250

[0083] If the 30,000 groups of data in the original data set are divided into training set and test set according to 8:2, 24,000 groups of training data and 6,000 groups of test data can be obtained. The training set is shown in Table 2 below. At this time, the training data includes the first training data with fault classification labels of 1, 2 and 3 respectively. Fault classification label 1 indicates that the fault classification is IR007, and the number of first sample signals of fault classification label 1 is 1000; fault classification label 2 indicates that the fault classification is B007, and the number of first sample signals of fault classification label 2 is 1000; fault classification label 3 indicates that the fault classification is OR007, and the number of first sample signals of fault classification label 3 is 1000. The training data also includes the second training data with the normal classification label of 0. The second training data includes three modes, namely: Normal-0hp (indicating that the working condition is 0 horsepower), Normal-1hp (indicating that the working condition is 1 horsepower) and Normal-2hp (indicating that the working condition is 2 horsepower), wherein the number of second sample signals of each mode is 7000.

[0084] Table 2:

[0085] Label category Modal Sample size 0 Normal Normal-0hp 7000 0 Normal Normal-1hp 7000 0 Normal Normal-2hp 7000 1 IR007 None 1000 2 B007 None 1000 3 OR007 None 1000

[0086] In other embodiments, the entire original data set may be used as a training set, and the test set required during the training process may be additionally collected. This embodiment does not limit the method for obtaining the training set. The original data set may be obtained by collecting signals in an actual scene to be classified, such as collecting sample signals and labeling the sample signals in a fault diagnosis scene; or it may be sent by other devices; this embodiment does not limit the method for obtaining the original data set.

[0087] Step 102: Divide the training set into at least two training subsets.

[0088] Each training subset includes various first training data and part of the second training data, and the second training data in different training subsets are different.

[0089] In this embodiment, by dividing each second training data into different training subsets, different training subsets reuse the first training data, so that subsequent intelligent agents can not only divide the learning of the larger-scale second training data to ensure the efficiency of reinforcement learning, but also make full use of the smaller-scale first training data to ensure the performance of reinforcement learning.

[0090] In one example, the second training data is multimodal data; accordingly, the training set is divided into at least two training subsets, including: dividing the second training data according to the modality of the second training data to obtain second training data corresponding to each modality; merging the second training data corresponding to each modality with each group of first training data to obtain a training subset corresponding to each modality.

[0091] Optionally, the multimodal data includes a second sample signal generated by a generation object of the second sample signal under different working conditions, and different working conditions correspond to different modes. Among them, the generation object of the second sample signal can be a device that generates the second sample signal, or it can also be a device that collects the second sample signal. This embodiment does not limit the implementation method of the generation object. The working condition refers to the working state of the generation object when generating the second sample signal. The same generation device can work under different working conditions. At this time, the generated second sample signal may be different. For example: the working condition is the rotational speed of the driving force, and the second sample signals generated at different rotational speeds may be different.

[0092] Accordingly, the second training data is divided according to the modality of the second training data to obtain the second training data corresponding to each modality, including: dividing the training data containing the second sample signals of the same modality into the same training subset.

[0093] For example, in the training set shown in Table 2 above, after the training set is divided according to the mode, the obtained training subsets are: training subset 1, training subset 2 and training subset 3. Among them, training subset 1 includes the second training data with the mode of Normal-0hp and all the first training data; training subset 2 includes the second training data with the mode of Normal-1hp and all the first training data; training subset 3 includes the second training data with the mode of Normal-2hp and all the first training data.

[0094] Table 3:

[0095] Training subset 1 Training subset 2 Training subset 3 Normal-0hp 7000 0 0 Normal-1hp 0 7000 0 Normal-2hp 0 0 7000 IR007 1000 1000 1000 B007 1000 1000 1000 OR007 1000 1000 1000

[0096] In other embodiments, the multimodal data may also be the second sample signal of multiple data domains obtained after converting the second sample signal of the first data domain into the second data domain, and different data domains correspond to different modes. For example: the second sample signal in the time domain is converted into the first time-frequency domain data by S transformation, and / or the second sample signal in the time domain is converted into the second time-frequency domain data by wavelet transformation, etc. At this time, the second sample signal in the time domain, the second sample signal of the first time-frequency domain data, and the second time-frequency domain data are respectively divided into different training subsets. At this time, the first sample signal of the first data domain can also be transformed into the second data domain by the same data domain transformation, and different training subsets reuse the first sample signals of all data domains; or, the first sample signal may not be converted in the data domain, and different training subsets reuse the first sample signal of the first data domain. This embodiment does not limit the implementation method of multimodal data.

[0097] In other embodiments, the second training data may be randomly and evenly divided, or divided into different training subsets in proportion. This embodiment does not limit the division method of the training set.

[0098] Step 103, for each training subset, reinforcement learning is performed on the training subset by the intelligent agent corresponding to the training subset to obtain local model parameters of the intelligent agent's reinforcement learning model after this round of training.

[0099] The model parameters of different reinforcement learning models before training are all global model parameters, and different training subsets correspond to different agents. Different agents refer to: different physical software programs that interact with the data environment, and the physical software program can be the process of running the agent.

[0100] Optionally, each agent performs reinforcement learning on the training subset in parallel to improve the training efficiency of the reinforcement learning model. At this time, for each training subset, the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training, including: creating multiple processes, where the processes correspond to the agents one by one; and running the corresponding agents in parallel through multiple processes to perform reinforcement learning on the training subset to obtain local model parameters obtained after this round of training.

[0101] In one example, multiple processes are created, including: using the torch.multiprocessing module in Python and using the spawn method to create multiple processes.

[0102] In Python, the torch.multiprocessing module provides multi-process functionality and can be used for parallel computing. The spawn method is a startup method in the torch.multiprocessing module that ensures that the code in the created different processes can run independently.

[0103] In another example, multiple processes are created, including: initializing a single-machine multi-process Ray cluster through the ray.init() function in Ray; defining a remote function through the @ray.remote decorator in Ray. Ray is an open source distributed framework that supports multi-process and distributed computing, and the ray.init() function in Ray is used to start multiple local processes and allow multiple tasks to be run concurrently. The @ray.remote decorator is used to declare a function or class as remotely callable, so that it can be executed in parallel on different nodes of the Ray cluster.

[0104] In other embodiments, the method for creating multiple processes on a single machine may be other methods, and this embodiment does not limit the method for creating multiple processes.

[0105] In other embodiments, the reinforcement learning of the training subset by the intelligent agent corresponding to the training subset may not be achieved through parallel training, but rather through sequential training according to a preset training order, or, one part is trained sequentially and the other part is trained in parallel. This embodiment does not limit the manner in which each intelligent agent performs reinforcement learning on the corresponding training subset.

[0106] Specifically, assume that the training set S is divided into three training subsets, namely: S1, S2, and S3. At this time, the training set can be expressed by the following formula:

[0107] S={S1,S2,S3}

[0108] Among them, S j represents the training subset corresponding to the agent running in the jth process, where j is less than or equal to 3. For each training subset S j , the training subset is reinforced by the agent corresponding to the training subset, and the local model parameters of the reinforcement learning model of the agent after this round of training are obtained, including the following steps 1031-1038:

[0109] Step 1031, for each training subset, generate a training sequence of each group of training data in the training subset in this round of training to obtain a data environment; wherein the training data includes the first training data and the second training data in the training subset.

[0110] Different rounds of training correspond to different training orders. Each training subset corresponds to a data environment. Optionally, before each round of training, the data environment will disrupt the order between different groups of training data in the training subset, use the disrupted order as training data, and return the sample signal in each group of training data to the agent as the current state.

[0111] Step 1032, return the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence through the data environment in accordance with the training order; the agent includes a reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer ranging from 1 to n, and n is the total number of training data in the training subset.

[0112] In one example, the network structure of the deep Q network in the reinforcement learning model of different agents is the same, that is, the same input layer, hidden layer and output layer are set, so that the model structure can be kept consistent in each process in a distributed environment, so that the training efficiency is basically the same. In addition, since different agents use different training subsets, the model parameters of the deep Q network in different agents may be different.

[0113] In other implementations, the network structures of the deep Q networks in different agents may also be different. This embodiment does not limit the model setting method of the deep Q networks in different agents.

[0114] In addition, the network structures of the deep Q network and the target network in the same agent are the same, but the model parameters can be different.

[0115] For example, the network structure of the deep Q network and the target network in the reinforcement learning model of different agents includes: two linear layers, which are set to 128 neurons and 32 neurons respectively, the number of neurons in the input layer is the dimension of the sample signal, such as 1200, and the number of neurons in the output layer is 4, representing the normal class and 3 types of fault classes, a total of four categories. In other embodiments, the network structure of the deep Q network and the target network can also be implemented in other ways, and this embodiment does not limit the implementation of the deep Q network and the target network.

[0116] In this step, the data environment inputs the i-th sample data in the i-th group of training data as the current state into the deep Q network in the agent. Initialization, i=1, that is, in the first time step of this round of training, the first sample data x1 in the first group of training data is input as the current state (or initial state) s1 into the deep Q network in the agent.

[0117] Step 1033, determining the i-th classification action corresponding to the i-th sample data through the deep Q network; wherein the i-th classification action indicates one of the first classification label and the second classification label.

[0118] In this embodiment, the action (i.e., the i-th classification action) performed by the agent based on the current state (i.e., the i-th sample data) is used to indicate one of the first classification label and the second classification label. Schematically, each classification label in the training set is pre-mapped to an action set A = {0, 1, ..., z}. Wherein, z is the total number of classification labels - 1. The action set A includes multiple different numerical values, each of which is used to indicate a classification label, and different numerical values ​​indicate different classification labels. For example: the value of z is 9, 0 represents the first classification label of normal classification, 1 to 9 all represent the second classification label of fault classification, and different numerical values ​​in 1 to 9 represent fault classification labels of different fault types. In other implementations, the numerical values ​​in A can also be other implementation methods, such as: excluding numerical values ​​​​of 0, etc. This embodiment does not limit the implementation method of the action set A.

[0119] In one example, in order to avoid the problem of falling into a local optimal solution when only using the deep Q network to determine the i-th classification action, the i-th classification action can be determined in combination with the ε greedy strategy. The deep Q network combined with the ε greedy strategy to determine the i-th classification action can be expressed as follows:

[0120]

[0121] Among them, a irepresents the i-th classification action; a represents a classification action randomly determined from the action set A; randomnumber is a random number generated by the data environment in the range of [0, 1]; ε represents the preset threshold in the data environment, 0<ε<1, which is used to measure the degree of exploration and utilization of the environment by the agent during the learning process; max a Q(s,a;θ) represents the maximum value of the Q values ​​corresponding to each classification action obtained by the deep Q network for the i-th sample data. a In Q(s,a;θ), a represents different classification actions, s represents the current state (i.e., the i-th sample data), θ is the current model parameter of the deep Q network, and Q represents the Q value corresponding to different classification actions output by the deep Q network. The above means that if the random number generated by the data environment in the range [0, 1] is less than ε, then the i-th classification action is randomly selected, and the i-th classification action is equal to max a Q(s,a;θ) is different. If the random number generated by the data environment in the range [0,1] is greater than or equal to ε, select max a Q(s,a;θ) is taken as the i-th classification action.

[0122] Step 1034, the intelligent agent returns the i-th classification action to the data environment, so that the data environment can determine the i-th reward of the i-th classification action according to the classification label corresponding to the i-th classification action and the sample data, and return the i-th reward and the i+1th sample data indicated by the training sequence to the intelligent agent.

[0123] In one example, the data environment determines the i-th reward of the i-th classification action through a reward function. Schematically, the reward function is represented by the following formula:

[0124]

[0125] Among them, r(s,a,s′) represents the reward when the current state is s, the next state is s', and the classification action corresponding to the current state is a; if the classification action is the same as the classification label corresponding to the current state s (i.e. right), then r(s,a,s′) is 1; if the classification action is different from the classification label corresponding to the current state s (i.e. wrong), then r(s,a,s′) is -1.

[0126] Step 1035, stores the i-th experience data through the experience pool in the agent, and updates i=i+1 to trigger the execution of the step of returning the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence through the data environment in the training order and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data.

[0127] Each set of i-th experience data stored in the experience pool can be expressed by the following formula:

[0128] B i =(s i , a i , r i ,s i+1 );

[0129] Among them, B i represents the i-th empirical data, s i represents the i-th sample data, a i represents the i-th classification action, r i represents the i-th reward, s i+1 Represents the i+1th sample data.

[0130] Assume that the state movement diagram of the agent at each time step in this round of training is as follows Figure 2 As shown, the data environment returns states s1, s2, ...s to the agent in the training order. n , after the deep Q network calculates each state separately, it obtains the classification action corresponding to the state. Figure 2 In this paper, we take z as 4 and a total of 5 categories as an example, and each classification action corresponds to a reward r.

[0131] Step 1036: After each set of training data in the training subset is traversed, K sets of experience data are randomly selected from the experience pool, where K is a positive integer.

[0132] Step 1037, input each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network.

[0133] Specifically, the weight and bias value parameters of the deep Q network are trained by the stochastic gradient descent method to minimize the difference between the first network result and the second network result of the target network, thereby maximizing the Q function value and obtaining the optimal strategy of the deep Q network. The difference between the first network result and the second network result of the target network is calculated by a preset loss function, which can be a mean square error cost function loss function established in combination with a reward. In other embodiments, the loss function can also be implemented in other ways, and this embodiment does not limit the implementation of the loss function.

[0134] Step 1038, each time W groups of experience data are traversed, the network parameters of the deep Q network are synchronized to the target network until each group of experience data is traversed, and the network parameters of the deep Q network obtained in this round of training are obtained; the network parameters of the deep Q network obtained in this round of training are the local model parameters obtained after this round of training. Wherein W is a positive integer.

[0135] An episode of training refers to the process in which the interaction between the agent and the data environment traverses each set of training data in the training subset and uses K sets of experience data in the experience pool to complete the training of the deep Q network.

[0136] In order to facilitate understanding of the process of obtaining local model parameters in this embodiment, refer to Figure 3 The interaction process between the intelligent agent and the data environment shown in Figure 3 It can be seen that in each process, when the agent corresponding to the process obtains the state s from the data environment, it decides on the action a. Then, the data environment returns the next state s' and the reward value r according to the action a. The experience pool (or replay memory unit) stores each experience data (or interaction data) in the replay memory unit B, and then randomly extracts K groups of experience data and inputs them into the deep Q network and the target network. The stochastic gradient descent method is used to train the weights and bias value parameters of the deep Q network to maximize the Q function value. After each W group of experience data is traversed, the network parameters of the deep Q network are synchronized to the target network until each group of experience data is traversed and the local model parameters of the deep Q network are obtained.

[0137] Step 104, based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training, and the global model parameters obtained in this round of training are used to assign values ​​to the model parameters of each reinforcement learning model before the next round of training.

[0138] In this embodiment, at the beginning of each round of training, the initialized local model parameters of each reinforcement learning model are global model parameters. For the first round of training, the initialized global model parameters are preset initial values. For training rounds after the first round of training, the global model parameters are parameter values ​​determined after that round of training based on the local model parameters of each reinforcement learning model.

[0139] In one example, based on the local model parameters corresponding to each reinforcement learning model, determining the global model parameters obtained in this round of training includes: obtaining the local model parameters corresponding to each reinforcement learning model obtained in this round of training; determining the average value of each local model parameter to obtain the global model parameter obtained in this round of training. At this time, the global model parameter can be calculated by the following formula:

[0140]

[0141] Among them, n represents the number of agents, that is, the number of training subsets. represents the local model parameters corresponding to the kth reinforcement learning model obtained in this round of training, W t+1 Represents the global model parameters corresponding to this round of training. The global model parameters are used to assign values ​​to the model parameters of each reinforcement learning model at the beginning of the next round of training.

[0142] In other implementations, the weighted average of each local model parameter may also be determined to obtain the global model parameter corresponding to the current round of training. This embodiment does not limit the method for determining the global model parameter.

[0143] In one example, obtaining local model parameters corresponding to each reinforcement learning model obtained in this round of training includes: reading the local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary, wherein the shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model.

[0144] For example, in a single machine with multiple processes, the multiprocessing library is used for inter-process communication, where shared memory is the key data transfer mechanism. Specifically, the share_memory() method is used to share global model parameters between processes, and the shared dictionary created by mp.Manager().dict() is used to save the network parameters trained by each process. In this way, each process can store the trained local model parameters in the shared dictionary, thus realizing inter-process data transfer.

[0145] Among them, the multiprocessing library is one of Python's standard libraries, allowing developers to use multiple processes to perform parallel tasks. The multiprocessing library provides a share_memory() method that allows processes to exchange data through shared memory. mp.Manager().dict() is another way to share resources in the multiprocessing library. It creates a server object in a child process, and other processes communicate with this server through the network to access and modify shared data. In this way, local model parameters can be accessed by accessing the shared dictionary created by mp.Manager().dict(), thereby calculating the local model parameters to obtain the global model parameters and storing them in the shared dictionary created by the share_memory() method.

[0146] Optionally, if the reinforcement learning model is trained synchronously between various processes, the Barrier mechanism can also be used to ensure that each process performs model training and parameter update operations synchronously at the appropriate time.

[0147] In another example, obtaining local model parameters corresponding to each reinforcement learning model obtained in this round of training includes: calling a pre-created remote function to obtain the local model parameters obtained by training the reinforcement learning model of each process; wherein different processes are used to train different reinforcement learning models.

[0148] For example: call a remote function and collect the results through the ray.get() function. Optionally, in stand-alone mode, you can also view process information, task execution status, etc. through the Dashboard.

[0149] In order to more clearly understand the process of determining the global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model, the update process of the global model parameters is illustrated by taking the average of the local model parameters corresponding to each reinforcement learning model to obtain the global model parameters. Figure 4 The input of the parameter averaging algorithm of the global model parameters is the initialized global model parameters W 0 , working node n (i.e., the process mentioned above), the threshold T of the training rounds; the output of the parameter averaging algorithm is the global model parameter W used in the end T For each round of training t∈[0, T-1], the current global model parameter W is broadcast to each working node through the master process Master. t , each working node k∈[1,n] obtains the global model parameter W t After that, as the initial model parameters of the reinforcement learning model corresponding to the working node, the reinforcement learning model is trained by the gradient descent method to obtain the local model parameters corresponding to this round of training The master process obtains the local model parameters obtained by each working node And update the global model parameters according to the following formula.

[0150]

[0151] In the case of training round t = T, the parameter averaging algorithm outputs W T .

[0152] Step 105, when the training rounds meet the preset conditions, determine the target reinforcement learning model based on the global model parameters obtained in the last round of training, and classify the input signal to be classified based on the target reinforcement learning model to obtain a classification result.

[0153] The target reinforcement learning model is determined based on the global model parameters obtained from the last round of training, including: using the network structure of the deep Q network in the agent and the global model parameters obtained from the last round of training to determine the target reinforcement learning model.

[0154] In one example, the training rounds satisfying the preset conditions include but are not limited to: the training rounds are equal to a preset training round threshold T; and / or the change in the loss function value after x consecutive rounds of training is less than a preset change threshold. This embodiment does not limit the implementation method of the preset conditions.

[0155] Optionally, after the model training is completed and the target reinforcement learning model is obtained, the target reinforcement learning model can also be tested using the test data in the test set to obtain the classification results corresponding to the test data, thereby testing the performance of the target reinforcement learning model.

[0156] In summary, the object classification method based on distributed multi-agent reinforcement learning provided in this embodiment obtains at least two training subsets by dividing the training set; wherein each training subset includes each first training data and part of the second training data; for each training subset, the training subset is reinforced learned by the agent corresponding to the training subset to obtain the local model parameters of the reinforcement learning model of the agent after this round of training, and the model parameters of different reinforcement learning models before training are all global model parameters; based on the local model parameters corresponding to each reinforcement learning model, the global model parameters obtained in this round of training are determined; when the training round meets the preset conditions, the target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to classify the input signal to be classified based on the target reinforcement learning model to obtain the classification result; the problem of low efficiency of traditional object classification methods can be solved. Since the target reinforcement learning model is obtained by distributed training based on parameter averaging, when classifying objects, only one target reinforcement learning model is used to realize object classification, without the need to be processed by the reinforcement learning models corresponding to multiple agents, thereby ensuring the efficiency of object classification. At the same time, it can solve the problem of limited classification ability of traditional object classification methods in data imbalance and multimodal scenarios. By reusing a small amount of first training data and having multiple agents divide the work of learning the second training data, the algorithm's ability to handle large-scale imbalanced data can be improved.

[0157] In addition, with the breakthrough of AlphaGo, deep reinforcement learning has become a recognized effective technology for solving continuous decision-making problems. Among them, AlphaGo is an intelligent robot that has defeated many professional Go players in human-computer games. At present, a large number of algorithms have been proposed to solve the challenging problems between deep reinforcement learning (DRL) and real-world applications, such as exploration and development dilemma, data inefficiency, multi-agent cooperation and competition, etc. Among all these challenges, data inefficiency is the most criticized problem because the trial-and-error learning mechanism of DRL requires a large amount of interactive data. In order to deal with this problem, inspired by distributed machine learning technology, distributed deep reinforcement learning (DDRL) has been proposed and successfully applied in the fields of computer vision and natural language processing. In recent years, in order to better train large-scale deep reinforcement learning algorithms, dedicated training frameworks have been proposed to support more complex real-time data sampling, model training and reasoning requirements by abstracting the algorithm interface and system design. Existing typical distributed reinforcement learning frameworks include Acme, Sample-Factory and SEEDRL, but these frameworks ignore the heterogeneity between multi-agent algorithms. There is no explicit support for learning paradigms that emphasize agent interaction, such as centralized training and collaborative algorithms based on communication functions, and there is a lack of corresponding unified interfaces to simplify algorithm implementation and training processes. In this embodiment, on the one hand, by providing a unified input and output interface for different types of multi-agent reinforcement learning algorithms, including standardized representations of information such as states, actions, and rewards, as well as a unified setting method for model structures and algorithm configuration parameters in different agents, algorithm interface standardization can be achieved; on the other hand, by setting up an inter-agent communication protocol, communication between different processes and model parameter sharing can be achieved; on the other hand, the same process can be set for development and testing of different links such as data preprocessing, model training, and model evaluation, which can simplify the development and testing difficulty of each link.

[0158] In addition, for the development of multi-agent reinforcement learning algorithm frameworks, existing work focuses more on algorithm implementation, and does not pay much attention to the scalability of the algorithm in large-scale scenarios, or more on algorithm libraries designed for certain scenarios, such as PyMARL and SMARTS frameworks. Therefore, for the scenarios supported by the multi-agent reinforcement learning algorithm framework, there has been a lack of a relatively comprehensive framework to implement the algorithm implementation, training, and deployment test process. In this embodiment, the framework of the object classification method can support algorithm training and testing in large-scale scenarios, including processing large-scale data sets, parallel training, etc. At the same time, the algorithm can further expand the data set, thereby supporting large-scale scenario applications. At the same time, the framework of the object classification method can easily integrate new multi-agent reinforcement learning algorithms, and support flexible configuration and modification of the algorithm, such as: increasing or decreasing the modality of the data, increasing or decreasing the configuration of agents, etc., to improve the scalability of the algorithm.

[0159] In order to more clearly understand the object classification method based on distributed multi-agent reinforcement learning proposed in this application, the object classification method is applied to a fault diagnosis scenario and the second training data includes three modes as an example for explanation. At this time, the target reinforcement learning model is used to perform fault diagnosis on the equipment; accordingly, the signal to be classified includes the equipment signal of the equipment; the first object classification includes fault signal classification, and the first classification label includes at least one fault classification label of the fault signal classification; the second object classification includes normal signal classification, and the second classification label includes normal classification label. Reference Figure 5 , the method includes the following steps:

[0160] Step 51, collection and division of data sets.

[0161] Assume that the CWRU dataset provided by Case Western Reserve University (CWRU) is used as the original dataset. In the CWRU dataset, there are differences in working conditions for the same type of equipment, resulting in differences between the monitored normal classification data, that is, there are different modes. Due to the huge amount of normal classification data, the agents can only exchange fault classification data, but not normal classification data. Taking the normal classification data under different horsepower as an example to simulate the normal classification data with different modes, the fault classification data that can be exchanged between multiple devices are introduced. A total of 30,000 groups of data were selected for the experiment, and the imbalance ratio of the overall data set (that is, the ratio of the number of normal classification data to the number of fault classification data) was 21:1. The specific data set description is shown in Table 1 above. The 30,000 samples are divided into training set and test set according to 8:2, of which the training set has 24,000 groups of training data, as shown in Table 2 above, and the test set has 6,000 groups of test data.

[0162] Step 52: Create a process corresponding to each modality and assign a training subset to each process.

[0163] According to the number of modes of the second training data in the training set, three agents are selected to interact with the data environment, and the 24,000 samples in the training set are reused by the first training data (i.e., the training data for fault classification) and the division of labor for learning different modes are used to allocate specific training subsets S1, S2, and S3 to the three agents respectively. The specific situation of the sample subset corresponding to each agent is shown in Table 3 above.

[0164] Step 53, reinforcement learning model construction.

[0165] Map each classification label in the training set to an action set A = {0, 1, 2, 3}. Among them, 0 represents the normal classification, 1 represents the IR007 fault type in the fault classification, 2 represents the B007 fault type in the fault classification, and 3 represents the OR007 fault type in the fault classification. The state space of the agent corresponding to each process is the sample data in the training subset corresponding to the process.

[0166] For each agent corresponding to each process, the state s of the agent at each time step t Will correspond to different sample data x in the training subset t At the beginning of training, the agent will randomly receive the first sample data x1 returned by the data environment as its initial state s1, and determine the corresponding classification action based on the initial state s1. After traversing all the sample data in the training subset, the training of one episode ends. When a new episode starts, the data environment will disrupt the order of the sample data in the training subset and interact with the agent again in the new order.

[0167] In each iteration, the agent corresponding to each process makes an action based on the data state returned by the data environment combined with the ε-greedy strategy. When the node corresponding to the action actually decided is consistent with the classification label corresponding to the sample data, the reward value is set to +1, otherwise the reward value is -1. The relevant description of the ε-greedy strategy and the reward function is detailed in the above embodiment, and this embodiment will not be repeated here.

[0168] Step 54: Initialize parameters and perform parallel training.

[0169] The reinforcement learning models corresponding to different processes are initialized as global model parameters and distributed to each process. Each process uses the experience data in the local experience pool for parallel training, updates the local model parameters through the reinforcement learning algorithm, and stores them in the shared dictionary.

[0170] Step 55: Parameter exchange and update.

[0171] After an episode, each process completes the training of the local reinforcement learning model. The local model parameters corresponding to each process obtained by the shared dictionary are averaged and used as the new global parameter model, and the updated global parameters are broadcast to each process. After a certain number of rounds of training, the trained deep Q network parameters are obtained, and the target reinforcement learning model is determined based on the deep Q network parameters.

[0172] Step 56: Model testing.

[0173] The test data in the test set is input into the target reinforcement learning model to obtain the classification results for model testing.

[0174] It has been verified that the object classification method based on distributed multi-agent reinforcement learning provided by this application has higher accuracy and classification performance. Specifically, taking CWRU as the original data set as an example, the data set includes normal classification data and fault classification data. The vibration data of the experiment is the acceleration data collected from the drive end (DE), with a sampling frequency of 12kHz. The bearing test bench device used to obtain the CWRU data set is as follows: Figure 6 As shown. The data set consists of nine types of fault classification data and one type of normal classification data. The fault types of fault classification include outer ring fault (OR), inner ring fault (IR) and roller fault (B), and the degree of fault also has different diameters, including 0.007mm, 0.014mm, and 0.021mm. The detailed information of fault classification and normal classification is shown in Table 4 below.

[0175] Table 4:

[0176] Label Fault Diameter(in.) Fault type Aliases 0 0 Normal Normal_097 1 0.007 IR IR007_105 2 0.007 B B007_118 3 0.007 OR OR007_130 4 0.014 IR IR014_169 5 0.014 B B014_185 6 0.014 OR OR014_197 7 0.021 IR IR021_209 8 0.021 B B021_222 9 0.021 OR OR021_234

[0177] In a possible implementation, in the scenario of solving fault diagnosis problems under data imbalance, the accuracy rate cannot fully reflect the performance of the model, and the F value is more suitable for reflecting the performance of the model under an unbalanced data set. Therefore, this embodiment introduces the F value as an evaluation indicator as an example. Before explaining the F value, the meanings of precision and recall are first introduced. The precision rate indicates the proportion of samples correctly predicted as positive examples by the classifier (that is, the target reinforcement learning model in this application), and the recall rate indicates the proportion of all samples that are actually positive examples that are correctly predicted as positive examples by the classifier. The precision rate focuses on the accuracy of the prediction results, and the recall rate focuses on the comprehensive coverage of positive examples. However, there is usually a trade-off between these two indicators, so the F value is introduced as a comprehensive evaluation indicator. The F value of classification c is denoted as F c , F c The calculation formula can be expressed as follows:

[0178]

[0179] Among them, Precision represents the precision of category c, and Recall represents the recall rate of category c; the higher the F value, the better the classifier performance. In multi-classification problems, the final F value calculation formula can be expressed as follows:

[0180]

[0181] Among them, C represents the total number of categories, F c Indicates the F value of the classification c.

[0182] Assume that the network structures of the deep Q network and the target network in each agent in the above embodiment include: two linear layers, set to 128 neurons and 32 neurons respectively, the number of neurons in the input layer is the dimension of the sample, that is, 1200, and the number of neurons in the output layer is 4, representing the normal class and 3 fault classes, a total of four categories, and the division method of the training subset can refer to the above Table 3. The parameter settings in deep reinforcement learning are as follows: ε in the ε greedy strategy = 0.1, the weight parameter δ in the loss function = 0.001, the number of experience data randomly selected from the experience pool K = 64, and the threshold episode of the training round = 20.

[0183] Take the single-agent reinforcement learning method and the group decision-making method based on multi-agent reinforcement learning as an example, and compare the object classification provided in this embodiment. Among them, the single-agent reinforcement learning method refers to the use of 24,000 training samples for reinforcement learning training, and the data imbalance ratio is 21:1; the idea of ​​allocating data sets by the group decision-making method based on multi-agent reinforcement learning is consistent with this article. Both allocate specific training set samples to each agent through fault sample reuse and multi-modal division of labor learning. There are 10,000 training samples for each agent, and the imbalance ratio is 7:1. However, in the training stage of the group decision-making method, each agent perceives the data features through a neural network, and learns strategies based on the reward value fed back by the data environment, so as to obtain the optimal strategy for each agent. In the final test stage, the samples are input into the models corresponding to each agent, and each agent generates its own classification results. The final classification results are obtained by voting and fuzzy integral group decision-making methods. The specific experimental results are shown in Table 5 below.

[0184] Table 5:

[0185]

[0186] According to Table 5, the method based on multi-agent reinforcement learning has higher accuracy and F value than the single-agent reinforcement learning method, which shows that the multi-agent system effectively alleviates the problems of data imbalance and multi-modality in the normal class through technologies such as fault sample reuse and multi-modal division of labor learning. By comparing the multi-agent reinforcement learning method, it is found that the distributed parameter averaging training method used in this application is better than the group decision method, with an accuracy increase of 2.25% and an F value increase of 0.0957. This shows that the method of allowing multiple agents to interact with each other during the training phase through parameter averaging in this application helps to improve the diagnostic effect of the algorithm.

[0187] In addition, Table VI shows the training time of single-agent reinforcement learning and distributed reinforcement learning methods.

[0188] Table 6:

[0189]

[0190] From a theoretical analysis, the distributed reinforcement learning method reuses fault samples, using 10,000 samples for training per process, while the single-agent reinforcement learning method uses 24,000 samples for training. Therefore, the efficiency of the distributed reinforcement learning method should be 24,000 / 10,000 = 2.4 times that of the single-agent reinforcement learning method. According to the results in Table 6, the efficiency of the distributed reinforcement learning method is 1.79 times that of the single-agent reinforcement learning method. This is because each process has to interact with parameters after each episode, which will sacrifice a certain amount of time, which is in line with reality. The experimental results in Table 6 further verify that the distributed training method based on parameter averaging in this application improves the efficiency of the algorithm.

[0191] In summary, the object classification method proposed in this application can solve the problems of data imbalance and multimodality of a certain type of data in object classification scenarios. It uses multi-agents to diagnose faults in the training set, and establishes an object classification method based on distributed multi-agent reinforcement learning through technologies such as fault sample reuse, division of labor learning for multimodality, and distributed training based on parameter averaging. It successfully overcomes the limitations of traditional single-agent algorithms in dealing with data imbalance and multimodality within normal categories, and significantly improves the algorithm's ability to process imbalanced data and the efficiency of processing large-scale imbalanced data.

[0192] Figure 7 7 is a block diagram of an object classification device based on distributed multi-agent reinforcement learning according to an exemplary embodiment. The device includes at least the following modules: a data acquisition module 710, a data partitioning module 720, a reinforcement learning module 730, a parameter updating module 740 and an object classification module 750.

[0193] The data acquisition module 710 is used to acquire a training set, wherein the training set includes multiple groups of first training data and multiple groups of second training data; wherein the number of the first training data is less than the number of the second training data; each group of the first training data includes a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of the second training data includes a second sample signal corresponding to a second object classification and a second classification label of the second object classification;

[0194] A data partitioning module 720 is used to partition the training set into at least two training subsets; wherein each training subset includes each first training data and a portion of the second training data, and the second training data in different training subsets are different;

[0195] A reinforcement learning module 730 is used to perform reinforcement learning on each training subset through the agent corresponding to the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training; model parameters of different reinforcement learning models before training are all global model parameters, and different training subsets correspond to different agents;

[0196] A parameter updating module 740 is used to determine the global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model, wherein the global model parameters obtained in this round of training are used to assign values ​​to the model parameters of each reinforcement learning model before the next round of training;

[0197] The object classification module 750 is used to determine the target reinforcement learning model based on the global model parameters obtained in the last round of training when the training rounds meet the preset conditions, so as to classify the input signal to be classified based on the target reinforcement learning model to obtain the classification result.

[0198] For detailed description, please refer to the above method embodiment.

[0199] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0200] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0201] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0202] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0203] Figure 8 1 is a block diagram of an object classification device 1900 based on distributed multi-agent reinforcement learning according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 8 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0204] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0205] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0206] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An object classification method based on distributed multi-agent reinforcement learning, characterized in that: The method comprises: Acquire a training set, the training set comprising a plurality of groups of first training data and a plurality of groups of second training data; wherein the number of the first training data is less than the number of the second training data; each group of the first training data comprises a first sample signal corresponding to a first object classification and a first classification label of the first object classification; each group of the second training data comprises a second sample signal corresponding to a second object classification and a second classification label of the second object classification; Dividing the training set into at least two training subsets; wherein each training subset includes each first training data and a portion of the second training data, and the second training data in different training subsets are different; For each training subset, the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain local model parameters of the agent's reinforcement learning model after this round of training; the model parameters of different reinforcement learning models before training are all global model parameters, and different training subsets correspond to different agents; Based on the local model parameters corresponding to each reinforcement learning model, determine the global model parameters obtained in this round of training, wherein the global model parameters obtained in this round of training are used to assign values ​​to the model parameters of each reinforcement learning model before the next round of training; When the training rounds meet the preset conditions, the target reinforcement learning model is determined based on the global model parameters obtained in the last round of training, so as to perform object classification on the input signal to be classified based on the target reinforcement learning model to obtain a classification result.

2. The method according to claim 1, characterized in that The second training data is multimodal data; accordingly, the training set is divided into at least two training subsets, including: Dividing the second training data according to the modality of the second training data to obtain second training data corresponding to each modality; The second training data corresponding to each modality is merged with each group of first training data to obtain a training subset corresponding to each modality.

3. The method according to claim 2, characterized in that The multimodal data includes second sample signals generated by a generating object of the second sample signal under different working conditions, and different working conditions correspond to different modes.

4. The method according to claim 1, characterized in that: For each training subset, the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training, including: Creating a plurality of processes, wherein the processes correspond one to one with the intelligent agents; The corresponding intelligent agents are run in parallel by the multiple processes to perform reinforcement learning on the training subset to obtain local model parameters after this round of training.

5. The method according to claim 1, characterized in that For each training subset, the agent corresponding to the training subset performs reinforcement learning on the training subset to obtain local model parameters of the reinforcement learning model of the agent after this round of training, including: For each training subset, a training sequence of each group of training data in the training subset in the current round of training is generated to obtain a data environment; wherein different training sequences correspond to different rounds of training; and the training data include first training data and second training data in the training subset; The data environment returns the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence according to the training order; the agent includes the reinforcement learning model, the reinforcement learning model includes a deep Q network and a target network, and the network parameters of the initialized deep Q network are the same as the network parameters of the target network; i is a positive integer taking values ​​from 1 to n in sequence, and n is the total number of training data in the training subset; Determining, by means of the deep Q network, an i-th classification action corresponding to the i-th sample data; wherein the i-th classification action indicates one of the first classification label and the second classification label; Returning the i-th classification action to the data environment through the agent, so that the data environment determines the i-th reward of the i-th classification action according to the classification label corresponding to the i-th classification action and the sample data, and returns the i-th reward and the i+1th sample data indicated by the training sequence to the agent; The i-th experience data is stored in the experience pool of the agent, and i=i+1 is updated to trigger the step of returning the i-th sample data in the i-th group of training data to the agent corresponding to the training subset in sequence through the data environment according to the training order and subsequent steps; the i-th experience data includes the i-th sample data, the i-th classification action, the i-th reward and the i+1-th sample data; After each set of training data in the training subset is traversed, K sets of experience data are randomly selected from the experience pool; K is a positive integer; Inputting each set of experience data into the deep Q network and the target network respectively, so as to update the deep Q network based on the difference between the first network result of the deep Q network and the second network result of the target network; Every time W groups of experience data are traversed, the network parameters of the deep Q network are synchronized to the target network until the traversal of each group of experience data is completed, and the network parameters of the deep Q network obtained in this round of training are obtained; the network parameters of the deep Q network obtained in this round of training are the local model parameters obtained after the current round of training, and W is a positive integer.

6. The method according to claim 1, characterized in that The determining of the global model parameters obtained in this round of training based on the local model parameters corresponding to each reinforcement learning model includes: Get the local model parameters corresponding to each reinforcement learning model obtained in this round of training; Determine the average value of each local model parameter to obtain the global model parameter obtained in this round of training.

7. The method according to claim 6, characterized in that The obtaining of local model parameters corresponding to each reinforcement learning model obtained in this round of training includes: Reading local model parameters corresponding to each reinforcement learning model from a pre-created shared dictionary, wherein the shared dictionary is used to store the local model parameters corresponding to each reinforcement learning model; or, Call a pre-created remote function to obtain local model parameters obtained by each process training the reinforcement learning model; different processes are used to train different reinforcement learning models.

8. The method according to any one of claims 1 to 7, characterized in that: The target reinforcement learning model is used to diagnose faults of the device; accordingly, the signal to be classified includes the device signal of the device; the first object classification includes fault signal classification, and the first classification label includes at least one fault classification label of the fault signal classification; the second object classification includes normal signal classification, and the second classification label includes a normal classification label.

9. An object classification device based on distributed multi-agent reinforcement learning, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images