A discrete manufacturing industry data representation method based on deep reinforcement learning

Abstract

The application discloses a kind of discrete manufacturing industry data representation methods based on deep reinforcement learning, including collecting discrete manufacturing industry data, construct space-time database;Discrete manufacturing industry data is divided into discrete features and continuous features, construct data coupling coding network, and the encoding vector in coding network is converted into representation vector, and construct data representation model;The distinguishing degree of the class of representation data is quantified by clustering evaluation index;Different dimensions of clustering evaluation index are weighted as dynamic reward, construct deep reinforcement learning model, update the neural network parameters of deep reinforcement learning through the interaction between representation model and discrete manufacturing decision analysis system.The application can represent dynamically changing mixed data, based on deep reinforcement learning, the discrete manufacturing industry data is represented and enhanced, interacts with discrete industrial system through clustering evaluation index, and constantly feedback dynamic reward information, to obtain the optimal data representation form.

Description

TECHNICAL FIELD

[0001] The application relates to a data representation method, in particular to a discrete manufacturing industry data representation method based on deep reinforcement learning. BACKGROUND

[0002] In the discrete manufacturing industry, data has the characteristics of multi-source heterogeneity, massiveness, high dimensionality, multi-scale, multi-time and space, etc., which leads to the fact that the traditional representation method cannot effectively process such data. Representing data through the similarity matrix of data objects is a common method. When learning the similarity, the coupling relationship between data can be integrated, such as ALGO, which describes the coupling relationship between feature values by calculating the conditional probability between feature values; COS comprehensively analyzes the coupling relationship between features and within features; and CMS proposes a distance measurement between data objects on the basis of COS. CURE captures the coupling relationship between feature values and features at different levels in the learning of data representation, and can be formulated into different representation algorithms according to different task schemes. The representation method of such data is generally suitable for discrete data.

[0003] However, the form of data in the discrete manufacturing industry is generally mixed data composed of continuous data and discrete data, and the representation method for mixed data is mostly obtained through data conversion, such as spectralCAT, which uses an automatic method to discretize continuous features, and constructs new discrete features through clustering of continuous features and class labels, but the converted data is applied to the clustering model as independent features, ignoring the relationship between different types of data features; CoupledMC uses similarity to represent discrete features on the basis of discretizing continuous features, but discretizing continuous variables causes information loss, so only using the Pearson correlation coefficient cannot well capture the relationship between continuous features and discrete features. With the development of deep learning technology, some researchers have applied deep learning to the field of discrete industrial manufacturing, but the existing deep learning algorithms are only applicable to static discrete industrial manufacturing environments, and are difficult to adapt to complex and dynamic discrete industrial manufacturing problems.

[0004] Deep reinforcement learning is an interactive learning method, and an intelligent agent learns through interaction with the environment, and can handle dynamic and complex environmental problems, so it is suitable for discrete industrial manufacturing problems. However, the rewards of the existing deep reinforcement learning algorithms are often set by artificial experience, and it is difficult to make the data representation decision provide the optimal solution for the discrete industrial manufacturing system. SUMMARY

[0005] The application aims to provide a discrete manufacturing industry data representation method based on deep reinforcement learning, which can represent mixed data that changes dynamically.

[0006] Technical solution: The present invention provides a discrete manufacturing industrial data representation method based on deep reinforcement learning, comprising the following steps:

[0007] (1) Collect discrete manufacturing industrial data and construct a spatiotemporal database;

[0008] (2) Divide discrete manufacturing industrial data into discrete features and continuous features, construct a data coupling coding network, and convert the coding vectors in the data coding network into representation vectors to construct a data representation model;

[0009] (3) The distinguishability of data categories is quantitatively represented by clustering evaluation indicators;

[0010] (4) The clustering evaluation indicators of different dimensions are weighted as dynamic rewards to construct a deep reinforcement learning model. The neural network parameters of deep reinforcement learning are updated by representing the interaction between the model and the discrete manufacturing decision analysis system.

[0011] Preferably, the discrete manufacturing industrial data in step (1) includes real-time workshop equipment data, APS production scheduling data, PDM product data, ERP inventory data, and MES production execution data.

[0012] Preferably, the construction of the data coupling coding network in step (2) includes: constructing a correlation matrix between discrete features and continuous features.

[0013]

[0014] in, It is a continuous feature; v j For discrete features; λ is the scaling factor; τ is the threshold parameter; Let be the joint probability density; and the expression for the calculation function of the joint probability density is:

[0015]

[0016] In the above formula, N is the number of data objects. discrete eigenvalues and v j Kernel functions between For kernel functions with continuous characteristics, Represent variable A i Continuous eigenvalues ​​f on the k-th data object i , Represent variable A i Continuous eigenvalues ​​f on the x-th data object i h i The bandwidth parameter represents the continuous feature; where the kernel function... The expression is:

[0017]

[0018] In the above formula, represents the discrete feature v j The corresponding feature value on the kth data object, λ is the proportional coefficient;

[0019] The correlation matrix is taken as the data coupling coding vector

[0020]

[0021] Coupling coding matrix M x The heterogeneous coupling relationship between the discrete feature and the continuous feature is represented, and the coupling coding matrix M x Quantized into the coding vector f.

[0022] Preferably, the step (2) of converting the coding vector in the data coding network into a representation vector comprises: using a fully connected network to convert the coding vector f into a representation vector

[0023] h = σ (f, W) in the formula, σ is a logistic function, W ∈ R contains the interaction intensity between all features.

[0024] Preferably, the deep reinforcement learning model in step (4) is DQN, and the Q router table is

[0025] Q' (s, a) = Q (s, a) + λ {R-Q {s, a)}

[0026] Wherein, Q (s, a) is the Q value of node s performing action a, wherein Q is the Q router table, s is the representation model node, a is the state action, λ represents the learning rate, R represents the reward information, Q' (s, a) represents the updated Q value, and Q (s, a) represents the Q value before updating.

[0027] Preferably, the reward information of the deep reinforcement learning in step (4) is dynamic reward

[0028]

[0029] Wherein, r i is a different dimension clustering evaluation index, α i is the weight coefficient of the different dimension clustering evaluation index, and R is the dynamic reward information.

[0030] Preferably, the different dimension clustering evaluation index includes CH index, DBI index and / or contour coefficient.

[0031] Preferably, the deep reinforcement learning model in step (4) further includes one of DDPG, A3C / A2C, PPO / TRPO, SAC, and TD3.

[0032] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: it can represent both mixed data and dynamically changing data, and uses deep reinforcement learning to enhance the representation of industrial big data in discrete manufacturing. It adopts a dynamic reward form, interacts with discrete industrial systems through clustering evaluation indicators, and continuously feeds back dynamic reward information to optimize the data representation dimension to the greatest extent and obtain the optimal data representation form. Attached Figure Description

[0033] Figure 1 This is a flowchart of the characterization method of the present invention;

[0034] Figure 2 This is a diagram of the deep reinforcement learning structure of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, the discrete manufacturing industrial data representation method based on deep reinforcement learning described in this invention includes the following steps:

[0037] (1) Collect discrete manufacturing industry data and construct a spatiotemporal database.

[0038] The collected discrete manufacturing industrial data includes real-time workshop equipment data, APS production scheduling data, PDM product data, ERP inventory data, and MES production execution data.

[0039] (2) Discrete manufacturing industrial data is divided into discrete features and continuous features, a data coupling coding network is constructed, and the coding vectors in the data coding network are converted into representation vectors to construct a data representation model. Specifically, this includes:

[0040] (2.1) Construct the correlation matrix between discrete features and continuous features

[0041]

[0042] in, It is a continuous feature; v j For discrete features; λ is the scaling factor; τ is the threshold parameter; Let be the joint probability density; and the expression for the calculation function of the joint probability density is:

[0043]

[0044] N is the number of data objects, is the discrete feature value and v j is the kernel function between them, is the kernel function of continuous feature, represents the variable A i the continuous feature value f i on the kth data object, represents the variable A i the continuous feature value f i on the xth data object, h i represents the bandwidth parameter of the continuous feature; wherein the kernel function is expressed as:

[0045]

[0046] In the above formula, represents the discrete feature v j corresponding feature value on the kth data object, and λ is the proportionality coefficient.

[0047] (2.2) Take the correlation matrix as the data coupling coding vector

[0048]

[0049] Coupling coding matrix M x characterizes the heterogeneous coupling relationship between discrete features and continuous features, and the coupling coding matrix M x is quantified as the coding vector f.

[0050] (2.3) Convert the coding vector f into the representation vector

[0051]

[0052] In the above formula, σ is the logistic function, W ∈ R contains the interaction intensity between all features.

[0053] (3) According to the needs of specific scenarios, select different dimensions of clustering evaluation indexes to quantify the discrimination degree of the representation data categories, wherein the clustering evaluation indexes include CH index, DBI index and silhouette coefficient;

[0054] Wherein the CH index is

[0055]

[0056] In the above formula, c i points to the ith category, n i points to c iThe number of data objects, d(x, y) represents the distance between data objects x and y.

[0057] The DBI index is

[0058]

[0059] In the above formula is the average Euclidean distance of the i-th data to its class center, and ||w i -w j ||2 is the Euclidean distance of the class centers of the i-th and j-th classes.

[0060] The contour coefficient is

[0061]

[0062] where i and j represent different intra-class sample points, and a(i) represents the cohesion of the sample point, that is, the similarity of the sample point to other points in the same cluster, and is calculated as follows:

[0063]

[0064] where distance represents the distance between i and j; b(i) represents the similarity of the sample point to other points in the next nearest cluster, and its calculation method is similar to a(i).

[0065] (4) According to the needs of different scenes, different dimensional clustering evaluation indexes are adjusted and weighted as dynamic rewards through weight coefficients, and a deep reinforcement learning model is constructed, and through the interaction between the representation model and the discrete manufacturing decision analysis system, the neural network parameters of the deep reinforcement learning are updated.

[0066] The deep reinforcement learning model adopts DQN, and its Q router table is

[0067] Q'(s, a) = Q(s, a) + λ{R-Q(s, a)}

[0068] Where Q(s, a) is the Q value of node s performing action a, where Q is the Q router table, s is the representation model node, a is the state action, λ is the learning rate, R is the reward information, Q'(s, a) is the updated Q value, and Q(s, a) is the Q value before updating. Its dynamic reward information R is

[0069]

[0070] Where α i is a parameter, and r iThe clustering evaluation index is for different dimensions. If the dynamic reward is maximized, the data representation is used for discrete manufacturing decision analysis system, if not, return to step (2), continuously feedback dynamic reward information, optimize the data representation dimension to the maximum extent, and obtain the optimal data representation form. The deep reinforcement learning model in step (4) is not limited to DQN, but also can adopt DDPG, A3C / A2C, PPO / TRPO, SAC and TD3 and the like.

Claims

1. A deep reinforcement learning based discrete manufacturing industry data representation method, characterized in that, The method comprises the following steps: (1) collecting discrete manufacturing industry data and constructing a space-time database; (2) dividing the discrete manufacturing industry data into discrete features and continuous features, constructing a data coupling coding network, converting the coding vectors in the data coding network into representation vectors, and constructing a data representation model; (3) quantifying the discrimination degree of the representation data categories through clustering evaluation indexes; (4) weighting different dimension clustering evaluation indexes as dynamic rewards to construct a deep reinforcement learning model, updating the neural network parameters of the deep reinforcement learning through the interaction between the representation model and the discrete manufacturing decision analysis system; if the dynamic reward is maximized, the representation data is used for the discrete manufacturing decision analysis system, if not, return to step (2), and continuously feedback the dynamic reward information to optimize the data representation dimension to the greatest extent to obtain the optimal data representation form.

2. The data representation method of claim 1, wherein, The discrete manufacturing industry data in step (1) includes real-time workshop equipment data, APS scheduling data, PDM product data, ERP inventory data, and MES production execution data.

3. The data representation method of claim 1, wherein, The step (2) of constructing the data-coupled coding network comprises: constructing a correlation matrix between discrete features and continuous features wherein, is a continuous feature; v j is a discrete feature; λ is a scale factor; τ is a threshold parameter; is a joint probability density; and the joint probability density is calculated by the function expression: In the above formula, N is the number of data objects, is a discrete feature value and v j is a kernel function between them, is a kernel function for continuous features, denotes the value of the variable A i on the kth data object for the continuous feature f i , denotes the value of the variable A i on the xth data object for the continuous feature f i , h i denotes the bandwidth parameter for the continuous feature; wherein the kernel function is expressed as: In the above formula, denotes the discrete feature v j corresponding feature value on the kth data object, and λ is a proportional coefficient. The correlation matrix is used as the data coupling coding vector Coupling coding matrix M x The heterogeneous coupling relationship between discrete features and continuous features is characterized, and the coupling coding matrix M x is quantified into a coding vector f.

4. The data representation method of claim 3, wherein, In step (2), the coding vector f in the data coding network is converted into a representation vector h by using a fully connected network h=σ(f,W) In the above formula, σ is a logistic function, W is a weight matrix, W ∈ R, R is a real number matrix, and contains the interaction strength between all features.

5. The data representation method of claim 1, wherein, In step (4), the deep reinforcement learning model is DQN, and the Q route table is Q'(s,a)=Q(s,a)+λ{R-Q(s,a)} Wherein, Q(s,a) is the Q value of node s performing action a, Q is the Q route table, s is the representation model node, a is the state action, λ represents the learning rate, R represents the reward information, Q'(s,a) represents the updated Q value, and Q(s,a) represents the Q value before updating.

6. The data representation method of claim 5, wherein, The reward information of the deep reinforcement learning in step (4) is dynamic reward wherein r i is a different dimension clustering evaluation index, a i is a different dimension clustering evaluation index weight coefficient, and R is a reward information.

7. The data representation method of claim 6, wherein, The different dimension clustering evaluation indexes include CH index, DBI index and / or contour coefficient.

8. The data representation method of claim 1, wherein, The deep reinforcement learning model in step (4) further comprises one of DDPG, A3C / A2C, PPO / TRPO, SAC and TD3.

Images