A reinforcement learning hybrid method applied to assembly sequence planning

Abstract

The application relates to a reinforcement learning hybrid method applied to assembly sequence planning, which can be understood as a kind of sorting of product parts in essence, and an excellent assembly sequence can shorten the production cycle of a product and realize the reduction of production cost. In the application, a reinforcement learning algorithm combining a Q-learning algorithm and a Sarsa algorithm, namely a Greedy-QS algorithm, is used to solve the assembly sequence planning problem. In the work of the application, a product to be assembled is reasonably abstracted into a mathematical model expressed by a plurality of Boolean matrices, the complexity of the assembly model is reduced, and the operability of the algorithm for the assembly sequence planning problem is improved, and then the Greedy-QS algorithm is used to obtain an optimal solution. Experiments prove that the Greedy-QS algorithm has a faster convergence speed than the Q-learning algorithm, the Sarsa algorithm and the Q-SARSA algorithm in the assembly sequence planning problem, and the reward value of the optimal assembly sequence obtained is higher, and the Greedy-QS algorithm has the advantages of the Q-learning algorithm and the Sarsa algorithm.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of assembly sequence planning, and particularly relates to a reinforcement learning method combining a Q-learning algorithm and a Sarsa algorithm. BACKGROUND

[0002] Assembly sequence planning (hereinafter referred to as ASP) plays an important role in modern manufacturing, which affects the process of how to effectively assemble unassembled parts into the final product. This problem not only concerns the improvement of production efficiency, but also directly affects product quality, production cost and delivery time. Solving the ASP problem needs to consider many factors, such as optimization of assembly sequence, assembly constraints between parts, and multi-objective optimization. Through accurate planning and advanced optimization methods, manufacturing enterprises can maximize production efficiency when facing the challenges of complex product manufacturing, and ensure that products meet high quality standards and can be delivered on time.

[0003] Reinforcement learning belongs to a category of machine learning, which is initially inspired by the learning method of biological systems. Reinforcement learning is a computational method that machines achieve goals through interaction with the environment. The agent concept is used to represent the decision-making machine. In reinforcement learning, the agent continuously optimizes its strategy by observing the environment state, selecting actions and receiving feedback (reward or punishment). The ultimate goal is to obtain the maximum cumulative reward expectation value in the process of multiple rounds of interaction. Reinforcement learning is very suitable for solving sequential decision-making problems. On this basis, we propose a reinforcement learning algorithm combining Q-learning algorithm and Sarsa algorithm. This algorithm adopts a left-right optimization strategy to integrate the above two algorithms and is named as greedy-QS algorithm.

[0004] After searching, the application publication number CN111222799B is a kind of assembly sequence planning method based on improved particle swarm algorithm. In the solving process of assembly problem, the particle swarm algorithm is easy to fall into local optimization, and multiple optimization solutions appear, but the actual feasible optimization solution efficiency is low. Therefore, according to the actual update of the global optimal value in the iteration process, the w parameter in the standard particle swarm algorithm is improved. Based on whether the change of the global optimal value before and after the last two times, the convergence accuracy and global search ability of the algorithm are improved by self-adaptive adjustment of the w parameter. The traditional standard w fixed mode is improved to a non-continuous adjustable parameter mode with iteration number as a variable under the consideration of the change of the global optimal value. The convergence accuracy and global search ability of the particle swarm algorithm can be effectively improved, and the problem of low efficiency of feasible solution in assembly sequence planning problem can be better solved.

[0005] The main content of the application is to use particle swarm algorithm to realize assembly sequence planning, the particle swarm algorithm belongs to one of heuristic algorithms, and has the shortcoming of easily falling into local optimization. The inventor adds different floating weights to the global optimal value obtained each time, so that the particle swarm algorithm jumps out of the local optimum. This method is undoubtedly effective but has certain limitations, for example, when the exploration space is too large, this method will still fall into a new optimal solution, and the exploration of the assembly sequence is not sufficient.

[0006] The algorithm used in the application is one of reinforcement learning algorithms, and the exploration of the assembly state is sufficient and not easy to fall into a local optimal solution by modifying the greedy strategy used therein. And two algorithms with complementary advantages are introduced in the application to solve the same model, so that the algorithm in the application not only has fast solving speed but also has better final solution. SUMMARY

[0007] The application aims to solve the problems of the prior art. A reinforcement learning hybrid method applied to assembly sequence planning is proposed. The technical scheme of the application is as follows:

[0008] A reinforcement learning hybrid method applied to assembly sequence planning, comprising the following steps:

[0009] Step S1, abstracting the assembly information of a model to be assembled, the assembly information including the number of parts, the contact condition between parts and the constraint condition between parts, and expressing the assembly information by using a Boolean matrix;

[0010] Step S2, building an applicable environment by using the information provided by the Boolean matrix, including the definition of state space, action space and reward function;

[0011] Step S3, alternately updating the Q function by using a greedy strategy to fuse the Q-learning algorithm (the core lies in learning a 'Q function', the expected cumulative reward obtained by taking a certain action in a given state) and the Sarsa algorithm (its name is derived from the first letters of five English words: State-Action-Reward-State-Action) to obtain a Greedy-QS algorithm;

[0012] Step S4, applying the environment model built in step S2 to the iteration logic of the Greedy-QS algorithm, and finally obtaining the most assembly sequence of the model to be assembled.

[0013] Further, the step S1 of expressing the assembly information by using a Boolean matrix specifically comprises the following steps:

[0014] First, the assembly parts are simplified into a series of assembly steps. Then, the contact relationships between these steps need to be described. Here, a Boolean matrix is ​​used to express these contact relationships; this matrix is ​​named the contact relationship matrix. As shown below, in the matrix, 'a'... ij The value is either 0 or 1; each row and column represents an assembly step, therefore the matrix has the same number of rows and columns, making it a square matrix; the values ​​in the matrix are related to whether contact occurs between assembly steps; 'a' in the matrix... ij Express the contact status between step i and step j; if contact exists, then a ij =1 otherwise the value is 0; it is worth adding that because there is no contact relationship in the same assembly step, naturally, the values ​​on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 = ... = 0 and this matrix is ​​a symmetric matrix, i.e., a 12 =a 21 ,a 23 =a 32 ,…;

[0015]

[0016] Secondly, the prohibited sequence of parts is generated by the constraints between assembly steps. A prohibited sequence means that there are some prohibited steps in the assembly process. If some assembly steps are assembled prematurely, the remaining assembly steps will be unable to be assembled, resulting in the failure of the assembly process. Therefore, steps containing prohibited sequences are not allowed in the assembly sequence planning. Here, a Boolean matrix is ​​used to express the prohibited sequence in the assembly process and it is named the prohibited sequence matrix, as shown below. The structure of the matrix is ​​similar to that of the matrix expressing contact, but the meaning of each row and column and the way the values ​​are expressed are different from the matrix above.

[0017]

[0018] In the forbidden sequence matrix, the number of rows and columns still represents the sequence number of each assembly step, but the difference is that the assembly step represented by the row number is the one that was assembled before the step represented by the column number.

[0019] Finally, the priority of the parts assembly is related to the assembly process of the parts. If the assembly of the parts is expected to be from left to right, the parts on the left are given a higher priority to guide the agent to find the optimal sequence. If the assembly of the parts is expected to be from bottom to top, the parts on the bottom are given a higher priority. Different levels are given different feedback values to play a guiding role. Similarly, the priority of the assembled parts can also be expressed by a Boolean matrix, which is named as the priority matrix, as shown below. Each row and column of the priority Boolean matrix has different meanings. The rows of the matrix represent the priority levels, which are defined according to different needs. The columns of the matrix represent the serial numbers of the parts, so the priority matrix is not a square matrix. Whether a value in the matrix is 1 or not determines the different levels of the parts.

[0020]

[0021] Further, in the step S2, the definition of the state space, the action space and the reward function is as follows:

[0022] First of all, the definition of the state space. In the specific context of the assembly sequence planning problem, the state space is related to the total number of parts of the model. If the model has n parts, the number of states that can occur during the assembly process is "2 n ". In order to build the environment in the algorithm, a binary digital representation is used to describe the state space of the model assembly.

[0023] In the overall environment of the assembly sequence planning problem, the action selected is the implementation of each assembly step. The size of the action space is equal to the number of parts to be assembled. The number of the state space plus the number of the action space can represent the next state.

[0024] Each round of assembly sequence conforms to the Markov decision process. By default, the assembly state is transferred n times under the condition that the assembly is successful each time. The action a selected each time is selected from the action space A. One assembly step can only occur once. At the same time of state transition, the action space should be reduced by the action selected in the previous step.

[0025] The reward function needs to be defined in combination with the contact relationship matrix of the model, the forbidden sequence matrix and the priority matrix. Here, the reward value triggered by the contact relationship matrix is defined as 4, the reward value triggered by the forbidden sequence planning is defined as -10, and the reward value triggered by the priority matrix is defined according to the difference between the levels of the parts before and after the action. If the next action is the same level as the previous action, the reward value is defined as 1. If the next action is one level lower than the previous action, the reward value is defined as 0.5. If the next action is two levels lower than the previous action, the reward value is defined as -3. If the next action is three levels lower than the previous action, the reward value is defined as -4.

[0026] Further, before updating the Q-value of the action in a certain state in the step S3, the updating formula of the Q-learning algorithm and the updating formula of the Sarsa algorithm are both used to make a calculation, and then the Q-value values obtained by the two are compared, and the one with a larger value is applied to this update.

[0027] Further, the Q-learning algorithm is specifically;

[0028] The Q function corresponding to the action in all states is initialized (usually zero or a small random value), an action a is selected according to the current state s and a strategy (such as an ∈-greedy strategy), the action a is executed, the reward r and the new state s' fed back by the environment are observed, the Q function is updated according to the updating formula of the Q-learning algorithm (the formula is shown below), and then the above actions are repeatedly performed until the algorithm converges.

[0029] The Sarsa algorithm is specifically;

[0030] The Q function corresponding to the action in all states is initialized (usually zero or a small random value), an action a is selected according to the current state s and a strategy (such as an ∈-greedy strategy), the action a is executed, the reward r and the new state s' fed back by the environment are observed, and then the next action a' is selected according to the same strategy using the new state s', and the Q function is updated according to the updating formula of the Sarsa algorithm (the formula is shown below), and then the above actions are repeatedly performed until the algorithm converges.

[0031] Further, the updating formula of the Q-learning algorithm and the updating formula of the Sarsa algorithm are respectively:

[0032] The updating formula of the Sarsa algorithm is:

[0033] Q(s t ,,a t )←Q(s t ,,a t )+α[r t +γQ(s t+1 ,,a t+1 )-Q(s t ,,a t )] (4)

[0034] The updating formula of the Q-learning algorithm is:

[0035] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)](5)

[0036] Further, the initial definition of the value of epsilon in step S3 is not in the range of 0 to 1, but takes a value greater than 1, and it is provided that the value of epsilon is gradually reduced each time the epsilon-greedy strategy is used until the desired value is reached.

[0037] An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the hybrid reinforcement learning method applied to assembly sequence planning when executing the program.

[0038] A non-transitory computer readable storage medium having stored thereon a computer program, the computer program implementing the hybrid reinforcement learning method applied to assembly sequence planning when executed by a processor.

[0039] Advantages and beneficial effects of the present application are as follows:

[0040] 1) The present application applies a hybrid reinforcement learning algorithm to assembly sequence planning problems, and the improved algorithm has fast convergence speed and excellent convergence results.

[0041] 2) In order to make the algorithm converge quickly, the value of epsilon in the epsilon-greedy strategy is gradually reduced with the progress of the algorithm rounds, effectively improving the convergence speed of the algorithm.

[0042] 3) The present application is easy to converge to a local optimal solution for complex assembly structure, and the value of epsilon is greater than 1, which greatly improves the exploration degree of the algorithm, and fully expands the Q-table zero algorithm in the early stage of the algorithm, which is not easy to fall into a local optimal solution.

[0043] The innovation of the present application lies in introducing two algorithms to apply to the same model, so that the advantages of the two algorithms can be complementary, thereby obtaining better results.

[0044] The innovation of the present application also lies in the innovative epsilon value in the greedy strategy being greater than 1, which can naturally expand the exploration ability of the algorithm, and continuously reduce the value of epsilon, so that the degree of freedom of the algorithm is gradually reduced, and finally the convergence of the algorithm is naturally realized. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the main flowchart of the preferred embodiment algorithm provided by the present application;

[0046] Figure 2 is the demonstration model of the stern tube provided by the present application;

[0047] Figure 3 is the effect diagram of the state transition in the algorithm of the present application;

[0048] Figure 4is the effect diagram of Q-learning algorithm in the same model and the same environment;

[0049] Figure 5 is the effect diagram of Sarsa algorithm in the same model and the same environment;

[0050] Figure 6 is the effect diagram of Q-Sarsa algorithm in the same model and the same environment;

[0051] Figure 7 is the effect diagram of Greedy-QS algorithm in the same model and the same environment. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only some of the embodiments of the present application.

[0053] The technical solutions of the present application to solve the above technical problems are:

[0054] Referring to Figure 1 , the present application proposes a flowchart of a reinforcement learning algorithm Greedy-QS algorithm combining Q-learning algorithm and Sarsa algorithm to solve the assembly sequence planning problem. The specific steps of the present embodiment are as follows:

[0055] Step S1, first, we simplify the assembly parts into some assembly steps, and then we describe the contact relationship between the assembly steps. Here, the contact relationship between the assembly steps is expressed in the form of a Boolean matrix, which is named as a contact relationship matrix. As shown below, the value of a ij in the matrix is 0 or 1. We define that each row and each column represents an assembly step, so the number of rows and columns of the matrix is equal, so the matrix is also a square matrix. Assuming that the assembly body is composed of 6 assembly steps, then the contact matrix expressing the contact relationship is 6 rows and 6 columns, that is, i=j=6 in the formula, and the same logic is used for analogy. The value in the matrix is related to whether the contact between the assembly steps occurs. The value of a ij in the matrix expresses the contact between step i and step j. If there is contact, then a ij =1, otherwise the value is 0. It is worth supplementing that because there is no contact relationship between the same assembly step, naturally, the values on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 =…=0, and the matrix is a symmetric matrix, that is, a 12 =a 21 , a 23 =a 32 , …

[0056]

[0057] Secondly, the forbidden sequence of parts is generated by the constraint relationship between assembly steps. The forbidden sequence means that there are some forbidden steps in the assembly process, that is, if some assembly steps are assembled in advance, the remaining assembly steps cannot be assembled, which leads to the failure of the assembly process. Therefore, the steps containing the forbidden sequence are not allowed to appear in the assembly sequence planning. Here, we can also use a Boolean matrix to express the forbidden sequence in the assembly process and name it as the forbidden sequence matrix, as shown below. The composition and expression of the matrix are similar to the above matrix, but the meaning of each row and column and the expression of the value are different from the above matrix.

[0058]

[0059] In the forbidden sequence matrix, the number of rows and columns still represents the sequence number of each assembly step, but the difference is that the row number represents the assembly step before the step represented by the column number. For example, a 12 represents that step 1 is assembled before step 2. If a 12 = 0 means that step 1 is allowed to be assembled before step 2, and if a 12 = 1 means that the sequence of step 1 before step 2 is a forbidden sequence. It is worth noting that a 12 represents all the sequences in which step 2 is arranged after step 1, not just the case where step 2 immediately follows step 1. Similarly, the values on the diagonal line in this matrix have no meaning, so a 11 = a 22 = a 33 = … = 0.

[0060] Finally, the priority of part assembly is related to the assembly process of the part. For example, if it is desired that the part assembly proceeds from left to right, a higher priority should be given to the left part to guide the agent to find the optimal sequence. If it is desired that the part assembly proceeds from bottom to top, a higher priority should be given to the lower part. Different levels give different feedback values to play a guiding role. Similarly, the priority of the assembled part can also be expressed by a Boolean matrix and named as the priority matrix, as shown below. Each row and column of the priority Boolean matrix also has different meanings. The row of the matrix represents the priority level, which is defined according to different needs. The column of the matrix represents the sequence number of the part, so the priority matrix is not a square matrix. We can determine the different levels of the part according to whether a 11 = 1 represents that part 1 is the highest level "1", and if a21 = 1, then the priority of part 1 is "2", and so on.

[0061]

[0062] In step S2, the information provided by the Boolean matrix builds the environment of the algorithm in the algorithm;

[0063] The Greedy-QS algorithm belongs to a kind of reinforcement learning algorithm, which needs state space, action space and reward function in the process. First of all, the definition of state space, in the specific context of assembly sequence planning problem, the state space is closely related to the total number of parts of the model. If the number of parts of the model is n, then the number of states that may appear in the assembly process is "2 n ". In order to build the environment in the algorithm, the binary digital expression is used to realize the description of the state space of model assembly. Here, the reference model is seen Figure Two , which can be divided into 11 assembly steps, so the state space is as follows:

[0064] Table 1 state space display

[0065]

[0066]

[0067]

[0068] In the environment of assembly sequence planning problem, the action selected is the implementation of each assembly step, and the size of the action space is equal to the number of parts to be assembled. Here we still assume that the assembly steps are 11, and the action space contains 11 elements. The action in the action space should make the state space of assembly transfer, and the binary number converted from the decimal number of state space represents assembly step 1 to assembly step 11 from left to right. Therefore, the action of implementing assembly step 1 is expressed as "10000000000", which corresponds to the decimal number 1024 (2 10 ), the action of implementing assembly step 2 is expressed as "01000000000", which corresponds to the decimal number 512 (2 9 ), and the rest is shown in the table below.

[0069] Decimal representation Assembly action a Binary representation 1024(2 10 )]]> Implementing assembly action 1 10000000000 512(2 9 )]]> Implementing assembly action 2 01000000000 256(2 8 )]]> Implementing assembly action 3 00100000000 128(2 7 )]]> Implementing assembly action 4 00010000000 64(2 6 )]]> Implementing assembly action 5 00001000000 32(2 5 )]]> Implementing assembly action 6 00000100000 16(2 4 )]]> Implementing assembly action 7 00000010000 8(2 3 )]]> Implementing assembly action 8 00000001000 4(2 2 )]]> Implementing assembly action 9 00000000100 2(2 1 )]]> Implementing assembly action 10 00000000010 1(2 0 )]]> Implementing assembly action 11 00000000001

[0070] In this way, the number of state space and the number of action space are added to get the number representing the next state, so the action space A = {2 10 , 2 9 , 2 8 , 2 7,2 6 ,2 5 ,2 4 ,2 3 ,2 2 ,2 1 ,2 0}。

[0071] Each round of assembly sequence is consistent with Markov decision process, and the assembly state is transferred 11 times by default, and the action a is selected from the action space A. It is worth noting that due to the particularity of the assembly sequence planning problem, because a assembly step can only occur once, the action space will be constantly reduced, and the action space should be reduced accordingly while the state is transferred.

[0072] The reward function needs to be defined in combination with the contact relationship matrix, the forbidden sequence matrix and the priority matrix of the model. Here, the reward value triggered by the contact relationship matrix is defined as 4, the reward value triggered by the forbidden sequence planning is defined as -10, and the reward value triggered by the priority matrix is defined according to the difference between the grades of the parts before and after. If the next action is the same grade as the previous action, the reward value is defined as 1, if the next action is one grade lower than the previous action, the reward value is defined as 0.5, if the next action is two grades lower than the previous action, the reward value is defined as -3, and if the next action is three grades lower than the previous action, the reward value is defined as -4. The change of the assembly model state is shown in Figure 3 .

[0073] In step S3, the Q-learning algorithm and the Sarsa algorithm are fused to obtain the Greedy-QS algorithm by a specific logic:

[0074] The main idea is to use the update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm to calculate once before updating the Q-value of the action in a certain state, and then compare the Q-value values obtained by the two, and use the larger one in the update.

[0075] The Sarsa algorithm update formula is:

[0076] Q(s t ,,a t )←Q(s t ,,a t )+α[r t +γQ(s t+1 ,,a t+1 )-Q(s t ,,a t )] (9)

[0077] The Q-learning algorithm update formula is:

[0078] Q(s, a)←Q(s, a) + a[r + y max a′ Q(s', a') - Q(s, a)] (10)

[0079] In addition to the above process, in order to make the algorithm have stronger exploration, a design is also made. When using the ε-greedy strategy, a random number with a size in the range of 0~1 is usually generated, and the number is compared with the ε value, if the number is less than ε, a random action is taken, otherwise the action value is selected according to the action. Here, the initial definition of the ε value is not in the range of 0~1, but a value greater than 1 is taken, and it is stipulated that the value of ε will gradually decrease every time the ε-greedy strategy is used, until the required value is reached. The advantage of this is that when ε is greater than 1, the agent will always take a random action, which can greatly enhance the exploration of the algorithm, and as the value of ε decreases, the algorithm will also converge, thereby obtaining the required solution.

[0080] In order to prove the advantage of the Greedy-QS algorithm provided by the application applied to the assembly sequence planning problem, the embodiment further compares the application effects of the Q-learning algorithm, the Sarsa algorithm and the Q-Sarsa algorithm in the assembly sequence planning problem to prove.

[0081] The same algorithm assembly environment uses the Greedy-QS algorithm, the Q-learning algorithm, the Sarsa algorithm and the Q-Sarsa algorithm to solve the optimal assembly sequence. The specific environment data is shown in Tables 1 and 2. The comparison results are shown in Figs. Figure 4 、 5 , 6, 7. It can be seen from the image that the convergence speed of the Greedy-QS algorithm is at an excellent level, and the result obtained is also the best among several algorithms, thereby verifying the superiority of the method.

[0082] The system, device, module or unit illustrated in the above embodiment can be specifically implemented by a computer chip or entity, or by a product with certain functions.

[0083] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0084] It should also be noted that the terms "comprising", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0085] The above embodiments should be understood as only for illustrating the present application and not for limiting the protection scope of the present application. After reading the content of the present application, the skilled in the art can make various changes or modifications to the present application, and these equivalent changes and modifications also fall within the scope defined by the claims of the present application.

Claims

1. A reinforcement learning hybrid method applied to assembly sequence planning, characterized in that, The method comprises the following steps: Step S1, abstracting assembly information of a model to be assembled for the model, the assembly information comprising part quantity, contact between parts and constraint between parts, and expressing the assembly information by using a Boolean matrix; Step S2, building an applicable environment by using information provided by the Boolean matrix, comprising definition of state space, action space and reward function; Step S3, alternately updating a Q function by using a greedy strategy to fuse a Greedy-QS algorithm; the core of the Q-learning algorithm is to learn a 'Q function', which is an expected cumulative reward obtained by taking a certain action in a given state; Sarsa is derived from the first letters of five English words: State-Action-Reward-State-Action algorithm; Step S4, applying the environment model built in step S2 to the iteration logic of the Greedy-QS algorithm, and finally obtaining an optimal assembly sequence of the model to be assembled by the converged algorithm; In step S3, before updating the Q-value of an action in a state, the updating formula of the Q-learning algorithm and the updating formula of the Sarsa algorithm are both used to calculate once, and then the Q-value values obtained by the two are compared, and the value with a larger one is applied to this update; In step S3, the initial definition of the ε value is not in the range of 0-1, but a value greater than 1, and it is specified that the value of ε will gradually decrease every time the ε-greedy strategy is used until the required value is reached.

2. The reinforcement learning hybrid method applied to assembly sequence planning according to claim 1, wherein, In step S1, the assembly information is expressed by using a Boolean matrix, which comprises the following steps: First, the assembly parts are simplified into a series of assembly steps. Then, the contact relationships between these steps need to be described. Here, a Boolean matrix is ​​used to express these contact relationships; this matrix is ​​named the contact relationship matrix. As shown below, in the matrix, 'a'... ij The value is either 0 or 1; each row and column represents an assembly step, therefore the matrix has the same number of rows and columns, making it a square matrix; the values ​​in the matrix are related to whether contact occurs between assembly steps; 'a' in the matrix... ij Express the contact status between step i and step j; if contact exists, then a ij =1 otherwise the value is 0; it is worth adding that because there is no contact relationship in the same assembly step, naturally, the values ​​on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 = ... = 0 and this matrix is ​​a symmetric matrix, i.e., a 12 =a 21 ,a 23 =a 32 ,…; Secondly, the forbidden sequence of parts is generated by the constraint relationship between assembly steps; the forbidden sequence means that there are some forbidden steps in the assembly process, that is, if some assembly steps are assembled in advance in the assembly process of the assembly steps, the remaining assembly steps cannot be assembled, which leads to the failure of the assembly process, therefore, the step of the forbidden sequence is not allowed to appear in the assembly sequence planning, and here, a forbidden sequence matrix is also used to express the forbidden sequence in the assembly process, which is named as a forbidden sequence matrix, as shown below; the composition and expression of the matrix are similar to those of the contact matrix, but the meaning represented by each row and column and the expression of the value are different from those of the above matrix; In the forbidden sequence matrix, the number of rows and columns still represents the serial number of each assembly step, but the difference is that the row number represents the assembly step before the step represented by the column number; Finally, the priority of the parts assembly is related to the assembly process of the parts. If the assembly of the parts is expected to be performed from left to right, the parts on the left are given a higher priority to guide the agent to find the optimal sequence. If the assembly of the parts is expected to be performed from bottom to top, the parts on the bottom are given a higher priority. Different levels are given different feedback values to play a guiding role. Similarly, the priority of the assembled parts can also be expressed by a Boolean matrix, which is named as a priority matrix, as shown below. Each row and column of the priority Boolean matrix has a different meaning. The rows of the matrix represent the priority levels, which are defined according to different needs. The columns of the matrix represent the serial numbers of the parts, so the priority matrix is not a square matrix. Whether a value in the matrix is 1 or not determines the different levels of the parts.

3. The reinforcement learning hybrid method applied to assembly sequence planning according to claim 1, wherein, In step S2, the definitions of the state space, the action space and the reward function are as follows: Firstly, the definition of state space, in the specific context of assembly sequence planning problem, the state space is related to the total number of parts of the model, if the model has the number of parts n, then the number of states that may appear in the assembly process is "2 n n", in order to build the environment in the algorithm, the binary digital expression is used to realize the description of the model assembly state space; Under the environment of assembly sequence planning, the selected action is the implementation of each assembly step. The size of the action space is equal to the number of parts to be assembled. The number of the state space is obtained by adding the number of the action space to the number of the state space. Each round of assembly sequence conforms to the Markov decision process. Under the condition that the assembly state is transferred n times each time, the selected action a is selected from the action space A. One assembly step can only occur once. At the same time of state transition, the action space should be reduced by the selected action of the previous step. The reward function needs to be defined in combination with the contact relationship matrix of the model, the forbidden sequence matrix and the priority matrix. Here, the reward value triggered by the contact relationship matrix is defined as 4, the reward value triggered by the forbidden sequence planning is defined as-10, and the reward value triggered by the priority matrix is defined according to the difference between the levels of the parts before and after the action. If the next action is of the same level as the previous action, the reward value is defined as 1. If the next action is one level lower than the previous action, the reward value is defined as 0.

5. If the next action is two levels lower than the previous action, the reward value is defined as-3. If the next action is three levels lower than the previous action, the reward value is defined as-4.

4. The reinforcement learning hybrid method applied to assembly sequence planning according to claim 1, wherein, The Q-learning algorithm is specifically as follows: The Q functions corresponding to the actions of all states are initialized. A action is selected according to the current state s and the strategy. The action a is executed. The reward r and the new state s' fed back by the environment are observed. Then the above actions are repeatedly performed until the algorithm converges. The Sarsa algorithm is specifically as follows: The Q functions corresponding to the actions of all states are initialized. A action is selected according to the current state s and the strategy. The action a is executed. The reward r and the new state s' fed back by the environment are observed. Different from the Q-learning algorithm, at this time, the next action a' is selected according to the same strategy based on the new state s'. The Q function is updated according to the update formula of the Sarsa algorithm. Then the above actions are repeatedly performed until the algorithm converges.

5. The reinforcement learning hybrid method applied to assembly sequence planning according to claim 1, wherein, The update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm are as follows: Sarsa algorithm update formula: Q(s,a)←Q(s,a)+α[r+γQ(s′,a′)-Q(s,a)](4) Q-learning algorithm update formula: Q(s, a)←Q(s, a) + a[r + y max a′ Q(s', a') - Q(s, a)] (5) In the formula: s represents the current state; a represents the action selected under the current state; s' represents the state after the action a is taken; a' represents the next action selected under the state s'; α represents the learning rate, which controls the step size of weight adjustment in each update process of the model; the learning rate determines the step size of the model parameters moving towards the optimal solution in each iteration; r represents the reward obtained by taking the action, which depends on the state s and the action a; γ represents the discount factor, which is used to balance the immediate reward and the future reward; the discount factor determines whether the algorithm should pay more attention to the immediate reward or the future reward when making decisions; Q(s, a) represents the Q value corresponding to the action a selected in the state s; Q(s', a') represents the Q value corresponding to the action a' selected in the state s'; max a′ Q(s',a') represents the maximum Q-value among all actions in state s'.

6. An electronic device, comprising: The computer program is executed by the processor to implement the reinforcement learning hybrid method for assembly sequence planning according to any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the reinforcement learning hybrid method for assembly sequence planning according to any one of claims 1 to 5.

Images