Reinforcement learning mixing method applied to assembly sequence planning

Abstract

The invention requests to protect a reinforcement learning mixing method applied to assembly sequence planning, the assembly sequence planning can be essentially understood as a sorting of product parts, the excellent assembly sequence can shorten the production cycle of products, and the production cost is reduced. In the invention, a reinforcement learning algorithm, i.e., a Greedy-QS algorithm, which combines a Q-learning algorithm and a Sarsa algorithm is provided to solve the problem of assembly sequence planning. In the work of the method, firstly, the to-be-assembled product is reasonably abstracted into a mathematical model expressed by several Boolean matrixes, the complexity of the assembly model is reduced, the operability of the algorithm for the assembly sequence planning problem is improved, and then the optimal solution is solved by using the Greedy-QS algorithm. Experiments prove that compared with a Q-learning algorithm, a Sarsa algorithm and a Q-SARSA algorithm, the Greedy-QS algorithm has a faster convergence speed in the assembly sequence planning problem, the obtained reward value of the optimal assembly sequence is higher, and the advantages of the Q-learning algorithm and the Sarsa algorithm are both achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of assembly sequence planning, and in particular relates to a reinforcement learning method combining a Q-learning algorithm and a Sarsa algorithm. Background Art

[0002] The assembly sequence planning (ASP) problem plays an important role in modern manufacturing. It affects the process of how to effectively assemble unassembled parts into final products. This problem is not only about improving production efficiency, but also directly affects product quality, production costs and delivery time. Solving the ASP problem requires comprehensive consideration of many factors, such as optimization of assembly sequence, assembly constraints between parts, and multi-objective optimization. Through precise planning and advanced optimization methods, manufacturing companies can maximize production efficiency when facing the challenges of complex product manufacturing, and ensure that products meet high quality standards and can be delivered on time.

[0003] Reinforcement learning is a category of machine learning, and was originally inspired by the learning methods of biological systems. Reinforcement learning is a computational method for machines to achieve goals by interacting with the environment. Reinforcement learning uses the concept of agent to represent the machine that makes decisions. In reinforcement learning, the agent continuously optimizes its own strategy by observing the state of the environment, choosing actions, and receiving feedback (rewards or penalties). The ultimate goal is to obtain the maximum expected value of cumulative rewards in multiple rounds of interaction. Reinforcement learning is very suitable for solving sequential decision-making problems. On this basis, we propose a reinforcement learning algorithm that combines the Q-learning algorithm and the Sarsa algorithm. This algorithm adopts a left-right best strategy to integrate the above two algorithms and is named the greedy-QS algorithm.

[0004] After searching, the application publication number CN111222799B is an assembly sequence planning method based on an improved particle swarm algorithm. This method takes into account that in the process of solving the assembly problem, the particle swarm algorithm is prone to fall into local optimization, resulting in multiple optimization solutions, but the actual feasible optimization solution efficiency is low. Therefore, according to the actual update of the global optimal value during the iteration process, the w parameter in the standard particle swarm algorithm is improved. Based on the change of the global optimal value of the previous and subsequent times, the convergence accuracy and global search ability of the algorithm are improved through adaptive adjustment of the w parameter. The traditional standard w fixed mode is improved to an adjustable parameter mode with discontinuous changes in the number of iterations as a variable under the consideration of the change of the global optimal value, which can effectively improve the convergence accuracy and global search ability of the particle swarm algorithm, and better solve the problem of low efficiency of feasible solutions in the assembly sequence planning problem.

[0005] The main content of this invention is to use the particle swarm algorithm to realize assembly sequence planning. The particle swarm algorithm is a kind of heuristic algorithm, which has the disadvantage of being easily trapped in local optimization. The inventor adds different floating weights to the global optimal value obtained each time, so that the particle swarm algorithm can jump out of the local optimality. This method is undoubtedly effective, but it has certain limitations. For example, when the exploration space is too large, this method will still fall into a new optimal solution, and the exploration of the assembly sequence is not sufficient.

[0006] The algorithm used in the present invention is a type of reinforcement learning algorithm, and by modifying the greedy strategy used therein, the algorithm is able to fully explore the assembly state and is not prone to falling into a local optimal solution. In addition, the present invention introduces two algorithms with complementary advantages to solve the same model, so that the algorithm in the present invention not only has a fast solution speed but also has a better final solution. Summary of the invention

[0007] The present invention aims to solve the above problems of the prior art. A reinforcement learning hybrid method applied to assembly sequence planning is proposed. The technical solution of the present invention is as follows:

[0008] A reinforcement learning hybrid method for assembly sequence planning, comprising the following steps:

[0009] Step S1, abstracting the assembly information of the model to be assembled, the assembly information including the number of parts, the contact conditions between parts and the constraints between parts, and expressing the assembly information using a Boolean matrix;

[0010] Step S2, building a suitable environment through the information provided by the Boolean matrix, including the definition of the state space, action space and reward function;

[0011] Step S3, the Q-learning (the core is to learn a 'Q function', the expected cumulative reward that can be obtained by taking a certain action in a given state) algorithm and the Sarsa (its name comes from the acronym of five English words: State-Action-Reward-State-Action) algorithm are alternately updated with a greedy strategy to obtain the Greedy-QS algorithm;

[0012] Step S4, applying the environment model constructed in step S2 to the iteration logic of the Greedy-QS algorithm, and finally the algorithm converges and obtains the optimal assembly sequence of the model to be assembled.

[0013] Furthermore, the step S1 of expressing the assembly information using a Boolean matrix specifically includes the following steps:

[0014] First, simplify the assembly parts into some assembly steps, and then describe the contact relationship between the assembly steps. Here, the contact relationship between the assembly steps is expressed in the form of a Boolean matrix, and this matrix is ​​named the contact relationship matrix; as shown below, the a in the matrix ij The value of is 0 or 1; each row and column represents an assembly step, so the number of rows and columns of the matrix is ​​equal, so this matrix is ​​also a square matrix; the value in the matrix is ​​related to whether contact occurs between assembly steps; a in the matrix ij Express the contact between step i and step j. If there is contact, then a ij =1 otherwise the value is 0; it is worth adding that because there is no contact relationship in the same assembly step, naturally, the values ​​on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 =…=0 and this matrix is ​​a symmetric matrix, i.e. a 12 =a 21 ,a 23 =a 32 ,…;

[0015]

[0016] Secondly, the forbidden sequence of parts is generated by the constraint relationship between assembly steps; the so-called forbidden sequence means that there are some forbidden steps in the assembly process, that is, if some assembly steps are assembled in advance during the assembly process, the remaining assembly steps will not be assembled, which will lead to the failure of the assembly process. Therefore, the steps that include forbidden sequences in the assembly sequence planning are not allowed to appear. Here, a Boolean matrix is ​​also used to express the forbidden sequence in the assembly process and it is named the forbidden sequence matrix, as shown below; here, the composition of the matrix is ​​similar to the matrix expressing contact, but the meaning of each row and column and the expression of the value are different from the above matrix;

[0017]

[0018] In the forbidden sequence matrix, the row and column numbers still represent the sequence number of each assembly step, but the difference is that the assembly step represented by the row number here is the step assembled before the step represented by the column number;

[0019] Finally, the priority of parts assembly is related to the assembly process of parts. If you want the parts to be assembled from left to right, you should give the parts on the left a higher priority to guide the agent to find the optimal sequence. If you want the parts to be assembled from bottom to top, you should give the parts below a higher priority. Different levels give different feedback values ​​to guide them. Similarly, the priority of assembling parts can also be expressed by a Boolean matrix and named as a priority matrix, as shown below. Each row and column of the priority Boolean matrix also has different meanings. The rows of the matrix represent the priority level, and different levels are defined according to different needs. The columns of the matrix represent the serial numbers of the parts, so the priority matrix is ​​not a square matrix. The different levels of the parts are judged based on whether a certain value in the matrix is ​​1.

[0020]

[0021] Furthermore, in step S2, the state space action space and reward function are defined as:

[0022] First, the definition of state space. In the specific context of assembly sequence planning, the state space is related to the total number of parts in the model. If the number of parts in the model is n, then the number of possible states in the assembly process is “2”. n ”,In order to build the environment in the algorithm, binary digital expression is used to achieve the description of the model assembly state space;

[0023] In the context of the assembly sequence planning problem, the action selected is the implementation of each assembly step. The size of the action space is equal to the number of parts to be assembled. The number in the state space is added to the number in the action space to get the number representing the next state.

[0024] Each round of assembly sequence conforms to the Markov decision process. By default, the assembly state is transferred n times if the assembly is successful every time. Each action a is selected from the action space A. An assembly step can only appear once. When the state is transferred, the action space should be subtracted from the action selected in the previous step.

[0025] It is necessary to define the reward function in combination with the contact relationship matrix, prohibited sequence matrix, and priority matrix of the model. Here, we define the reward value for triggering the contact relationship matrix as 4, the reward value for triggering the prohibited sequence planning as -10, and the situation of triggering the priority matrix is ​​determined according to the difference in levels between the front and rear parts. If the latter action is at the same level as the previous action, the reward value is defined as 1. If the latter action is one level lower than the previous action, the reward value is defined as 0.5, the reward value is defined as -3 for two levels lower, and the reward value is defined as -4 for three levels lower.

[0026] Furthermore, in step S3, before the Q-value of an action in a certain state is updated, both the update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm are used to perform a calculation in advance, and then the Q-values ​​obtained by the two are compared, and the one with the larger value is used for this update.

[0027] Furthermore, the Q-learning algorithm is specifically:

[0028] Initialize the Q function corresponding to the actions in all states (usually zero or a small random value), select an action a according to the current state s and strategy (such as ∈-greedy strategy), execute action a, observe the reward r of the environment feedback and the new state s′ transferred to, update the Q function according to the update formula of the Q-learning algorithm (see the formula below), and then repeat the above actions until the algorithm converges.

[0029] The Sarsa algorithm is specifically:

[0030] Initialize the Q function corresponding to the actions in all states (usually zero or a small random value), select an action a according to the current state s and strategy (such as ∈-greedy strategy), execute action a, observe the reward r of the environment feedback and the new state s′ transferred to. Different from the Q-learning algorithm, at this time, the next action a′ is selected according to the new state s′ using the same strategy, and the Q function is updated according to the update formula of the Sarsa algorithm (see the formula below), and then the above actions are repeated until the algorithm converges.

[0031] Furthermore, the update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm are respectively:

[0032] Sarsa algorithm update formula:

[0033] Q(s t ,,a t )←Q(s t ,,a t )+α[r t +γQ(s t+1 ,,a t+1 )-Q(s t ,,a t )] (4)

[0034] Q-learning algorithm update formula:

[0035] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)](5)

[0036] Furthermore, in step S3, the initial definition of the ε value is not in the range of 0 to 1, but a value greater than 1, and it is stipulated that each time the ε-greedy strategy is used, the ε value will be gradually reduced until the required value is reached.

[0037] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the reinforcement learning hybrid method for assembly sequence planning as described in any one of the items is implemented.

[0038] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any one of the reinforcement learning hybrid methods for assembly sequence planning.

[0039] The advantages and beneficial effects of the present invention are as follows:

[0040] 1) The present invention utilizes a hybrid reinforcement learning algorithm to apply to the assembly sequence planning problem. The improved algorithm has a fast convergence speed and excellent convergence results.

[0041] 2) In order to make the algorithm converge quickly, the present invention designs the ε value in the ε-greedy strategy to gradually decrease as the algorithm rounds proceed, which effectively improves the convergence speed of the algorithm.

[0042] 3) The present invention aims at complex assembly structures and easily converges to obtain a local optimal solution in advance, making the ε value greater than 1, which greatly improves the exploration degree of the algorithm and fully expands the Q-table zero algorithm in the early stage of the algorithm so that it is not easy to fall into the local optimal solution.

[0043] The innovation of the present invention is to introduce two algorithms and apply them to the same model, so that the advantages of the two algorithms can complement each other and obtain better results.

[0044] The innovation of the present invention also lies in the innovative setting of the ε value in the greedy strategy to be greater than 1, which can naturally expand the exploration ability of the algorithm and continuously reduce the ε value, so that the degree of freedom of the algorithm is gradually reduced and the convergence of the algorithm is naturally achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the main flow chart of the algorithm of the preferred embodiment provided by the present invention;

[0046] Figure 2 A demonstration model stern tube for use of the present invention;

[0047] Figure 3 It is an effect diagram of the state transition in the algorithm of the present invention;

[0048] Figure 4This is a diagram of the effect of the Q-learning algorithm under the same environment with the same model;

[0049] Figure 5 This is the effect diagram of the Sarsa algorithm under the same environment of the same model;

[0050] Figure 6 This is the effect diagram of the Q-Sarsa algorithm under the same environment of the same model;

[0051] Figure 7 This is a diagram showing the effect of the Greedy-QS algorithm in the present invention under the same model and the same environment. DETAILED DESCRIPTION

[0052] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0053] The technical solution of the present invention to solve the above technical problems is:

[0054] See also Figure 1 , the present invention proposes a reinforcement learning algorithm Greedy-QS algorithm that combines Q-learning algorithm and Sarsa algorithm to solve the flow chart of the assembly sequence planning problem. The specific steps of this embodiment are as follows:

[0055] Step S1, first, we simplify the assembly parts into some assembly steps, and then we need to describe the contact relationship between the assembly steps. Here, we use the Boolean matrix to express the contact relationship between the assembly steps. We name this matrix the contact relationship matrix. As shown below, the matrix a ij The value of is 0 or 1. We define each row and column to represent an assembly step, so the number of rows and columns of the matrix is ​​equal, so this matrix is ​​also a square matrix. Assuming that the assembly is composed of 6 assembly steps, then the contact matrix expressing the contact relationship is 6 rows and 6 columns, that is, i=j=6 in the formula, and the same logic is used. The value in the matrix is ​​related to whether contact occurs between assembly steps. The a in the matrix ij Express the contact between step i and step j. If there is contact, then a ij =1 otherwise the value is 0. It is worth adding that since there is no contact relationship in the same assembly step, naturally, the values ​​on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 =…=0 and this matrix is ​​a symmetric matrix, i.e. a 12 =a 21 ,a 23 =a 32 ,….

[0056]

[0057] Secondly, the prohibited sequence of parts is generated by the constraint relationship between assembly steps. The so-called prohibited sequence means that there are some prohibited steps in the assembly process, that is, if some assembly steps are assembled in advance during the assembly process, the remaining assembly steps will not be assembled, which will lead to the failure of the assembly process. Therefore, the steps that include prohibited sequences in the assembly sequence planning are not allowed to appear. Here we can also use a Boolean matrix to express the prohibited sequence in the assembly process and name it the prohibited sequence matrix, as shown below. The composition of the matrix here is similar to the matrix expressing contact, but the meaning of each row and column and the expression of the value are different from the above matrix.

[0058]

[0059] In the forbidden sequence matrix, the row and column numbers still represent the sequence number of each assembly step, but the difference is that the assembly step represented by the row number here is the step that is assembled before the step represented by the column number. 12 It means that step 1 is assembled before step 2. If a 12 = 0 means that step 1 is allowed to be assembled before step 2, if a 12 = 1 means that the sequence in which step 1 comes before step 2 is a forbidden sequence. It is worth noting that a 12 represents all sequences where step 2 comes after step 1, rather than just the case where step 2 comes immediately after step 1. Similarly, the values ​​on the diagonal in this matrix are meaningless, so a 11 =a 22 =a 33 =…=0.

[0060] Finally, the priority of parts assembly is related to the assembly process of parts. For example, if you want the parts to be assembled from left to right, you must give the parts on the left a higher priority to guide the agent to find the optimal sequence. If you want the parts to be assembled from bottom to top, you must give the parts below a higher priority. Different levels give different feedback values ​​to guide them. Similarly, the priority of assembling parts can also be expressed using a Boolean matrix and named the priority matrix, as shown below. Each row and column of the priority Boolean matrix also has different meanings. The rows of the matrix represent the priority level, and different levels are defined according to different needs. The columns of the matrix represent the serial numbers of the parts, so the priority matrix is ​​not a square matrix. We judge the different levels of parts based on whether a value in the matrix is ​​1. For example, if a 11 =1 means that part 1 is the highest level "1", if a21 =1 means that the priority of part 1 is "2", and so on.

[0061]

[0062] In step S2, an environment suitable for the algorithm is built in the algorithm through the information provided by the Boolean matrix;

[0063] Greedy-QS algorithm is a type of reinforcement learning algorithm. It requires state space, action space and reward function in the process. First of all, the definition of state space. In the specific context of assembly sequence planning problem, the state space is closely related to the total number of parts of the model. If the number of parts in the model is n, then the number of states that may appear in the assembly process is "2 n In order to build the environment in the algorithm, binary digital expression is used to describe the model assembly state space. Figure 2 , the model can be divided into 11 assembly steps, so the state space is as follows:

[0064] Table 1 State space display

[0065]

[0066]

[0067]

[0068] In the context of assembly sequence planning, the action selected is the implementation of each assembly step. The size of the action space is equal to the number of parts to be assembled. Here we still assume that there are 11 assembly steps, and the action space contains 11 elements. The actions in the action space must allow the state space of the assembly to be transferred. As mentioned above, the binary numbers converted from the decimal numbers in the state space represent assembly steps 1 to 11 from the left. Therefore, the action to implement assembly step 1 is expressed as "10000000000", which corresponds to the decimal number 1024 (2 10 ), the action of implementing assembly step 2 is expressed as "01000000000" corresponding to the decimal number 512 (2 9 ), and the rest are detailed in the table below.

[0069] Decimal expression Assembly action a Binary Expression <![CDATA[1024(2 10 )]]> Implement assembly action 1 10000000000 <![CDATA[512(2 9 )]]> Implement assembly action 2 01000000000 <![CDATA[256(2 8 )]]> Implement assembly action 3 00100000000 <![CDATA[128(2 7 )]]> Implement assembly action 4 00010000000 <![CDATA[64(2 6 )]]> Implement assembly action 5 00001000000 <![CDATA[32(2 5 )]]> Perform assembly action 6 00000100000 <![CDATA[16(2 4 )]]> Implement assembly action 7 00000010000 <![CDATA[8(2 3 )]]> Implement assembly action 8 00000001000 <![CDATA[4(2 2 )]]> Perform assembly action 9 00000000100 <![CDATA[2(2 1 )]]> Perform assembly action 10 00000000010 <![CDATA[1(2 0 )]]> Implement assembly action 11 00000000001

[0070] With this design, the sum of the numbers in the state space and the numbers in the action space can give the number representing the next state, so the action space A is defined as {2 10 ,2 9 ,2 8 ,2 7,2 6 ,2 5 ,2 4 ,2 3 ,2 2 ,2 1 ,2 0}.

[0071] Each round of assembly sequence conforms to the Markov decision process. By default, the assembly state is transferred 11 times, and the action a selected each time is selected from the action space A. It is worth noting that due to the particularity of the assembly sequence planning problem, since an assembly step can only appear once, the action space here will continue to shrink. When the state is transferred, the action space should be subtracted from the action selected in the previous step.

[0072] It is necessary to define the reward function in combination with the contact relationship matrix, forbidden sequence matrix, and priority matrix of the model. Here, we define the reward value for triggering the contact relationship matrix as 4, the reward value for triggering the forbidden sequence planning as -10, and the situation of triggering the priority matrix is ​​determined according to the difference in the levels of the front and rear parts. If the latter action is at the same level as the previous action, the reward value is defined as 1. If the latter action is one level lower than the previous action, the reward value is defined as 0.5, the reward value is defined as -3 for two levels lower, and the reward value is defined as -4 for three levels lower. The changes in the assembly model state are as follows: Figure 3 shown.

[0073] In step S3, the Q-learning algorithm and the Sarsa algorithm are fused with specific logic to obtain the Greedy-QS algorithm:

[0074] The main idea is to use the Q-learning algorithm’s update formula and the Sarsa algorithm’s update formula to calculate the Q-value of an action in a certain state before updating it. Then compare the Q-values ​​they both give, and use the larger one for the update.

[0075] Sarsa algorithm update formula:

[0076] Q(s t ,,a t )←Q(s t ,,a t )+α[r t +γQ(s t+1 ,,a t+1 )-Q(s t ,,a t )] (9)

[0077] Q-learning algorithm update formula:

[0078] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)](10)

[0079] In addition to the above process, a design was made to make the algorithm more exploratory. When using the ε-greedy strategy, a random number between 0 and 1 is usually generated, and this number is compared with the ε value. If this number is less than ε, a random action will be taken, otherwise an action will be selected based on the action value. Here, our initial definition of the ε value is not between 0 and 1, but a value greater than 1, and it is stipulated that each time the ε-greedy strategy is used, the ε value will gradually decrease until it reaches the required value. The advantage of this is that when ε is greater than 1, the agent will always take random actions, which can greatly enhance the exploratory nature of the algorithm, and as the ε value decreases, the algorithm will converge accordingly, thereby obtaining the required solution.

[0080] In order to prove the advantages of the Greedy-QS algorithm provided by the present invention in applying to the assembly sequence planning problem, this embodiment further proves it by comparing the application effects of the Greedy-QS algorithm, the Sarsa algorithm, and the Q-Sarsa algorithm in the assembly sequence planning problem.

[0081] The Greedy-QS algorithm, Q-learning algorithm, Sarsa algorithm, and Q-Sarsa algorithm are used in the same algorithm assembly environment to solve the optimal assembly sequence. The specific environment data are shown in Table 1 and Table 2. The comparison results are shown in Figure 4 , 5 , 6, and 7. It can be seen from the images that the convergence speed of the Greedy-QS algorithm is at an excellent level, and the results obtained are also the best among several algorithms, which verifies the superiority of this method.

[0082] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0083] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0085] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A reinforcement learning hybrid method for assembly sequence planning, characterized in that: The following steps are involved: Step S1, abstracting the assembly information of the model to be assembled, the assembly information including the number of parts, the contact conditions between parts and the constraints between parts, and expressing the assembly information using a Boolean matrix; Step S2, building a suitable environment through the information provided by the Boolean matrix, including the definition of the state space, action space and reward function; Step S3, the Q-learning algorithm and the Sarsa algorithm are alternately updated with a greedy strategy to obtain the Greedy-QS greedy-QS algorithm; the core of the Q-learning algorithm is to learn a 'Q function', which is the expected cumulative reward that can be obtained by taking a certain action under a given state; the name Sarsa comes from the acronym of five English words: State-Action-Reward-State-Action algorithm; Step S4, applying the environment model constructed in step S2 to the iteration logic of the Greedy-QS algorithm, and finally the algorithm converges and obtains the optimal assembly sequence of the model to be assembled.

2. A reinforcement learning hybrid method for assembly sequence planning according to claim 1, characterized in that: The step S1 uses a Boolean matrix to express the assembly information, and specifically includes the following steps: First, simplify the assembly parts into some assembly steps, and then describe the contact relationship between the assembly steps. Here, the contact relationship between the assembly steps is expressed in the form of a Boolean matrix, and this matrix is ​​named the contact relationship matrix; as shown below, the a in the matrix ij The value of is 0 or 1; each row and column represents an assembly step, so the number of rows and columns of the matrix is ​​equal, so this matrix is ​​also a square matrix; the value in the matrix is ​​related to whether contact occurs between assembly steps; a in the matrix ij Express the contact between step i and step j. If there is contact, then a ij =1 otherwise the value is 0; it is worth adding that because there is no contact relationship in the same assembly step, naturally, the values ​​on the diagonal of the matrix are all 0, that is, a 11 =a 22 =a 33 =…=0 and this matrix is ​​a symmetric matrix, i.e. a 12 =a 21 ,a 23 =a 32 ,…; Secondly, the forbidden sequence of parts is generated by the constraint relationship between assembly steps; the so-called forbidden sequence means that there are some forbidden steps in the assembly process, that is, if some assembly steps are assembled in advance during the assembly process, the remaining assembly steps will not be assembled, which will lead to the failure of the assembly process. Therefore, the steps that include forbidden sequences in the assembly sequence planning are not allowed to appear. Here, a Boolean matrix is ​​also used to express the forbidden sequence in the assembly process and it is named the forbidden sequence matrix, as shown below; here, the composition of the matrix is ​​similar to the matrix expressing contact, but the meaning of each row and column and the expression of the value are different from the above matrix; In the forbidden sequence matrix, the row and column numbers still represent the sequence number of each assembly step, but the difference is that the assembly step represented by the row number here is the step assembled before the step represented by the column number; Finally, the priority of parts assembly is related to the assembly process of parts. If you want the parts to be assembled from left to right, you should give the parts on the left a higher priority to guide the agent to find the optimal sequence. If you want the parts to be assembled from bottom to top, you should give the parts below a higher priority. Different levels give different feedback values ​​to guide them. Similarly, the priority of assembling parts can also be expressed by a Boolean matrix and named as a priority matrix, as shown below. Each row and column of the priority Boolean matrix also has different meanings. The rows of the matrix represent the priority level, and different levels are defined according to different needs. The columns of the matrix represent the serial numbers of the parts, so the priority matrix is ​​not a square matrix. The different levels of the parts are judged based on whether a certain value in the matrix is ​​1.

3. The reinforcement learning hybrid method for assembly sequence planning according to claim 1, characterized in that: In step S2, the state space action space and reward function are defined as: First, the definition of state space. In the specific context of assembly sequence planning, the state space is related to the total number of parts in the model. If the number of parts in the model is n, then the number of possible states in the assembly process is "2 n ”,In order to build the environment in the algorithm, binary digital expression is used to achieve the description of the model assembly state space; In the context of the assembly sequence planning problem, the action selected is the implementation of each assembly step. The size of the action space is equal to the number of parts to be assembled. The number in the state space is added to the number in the action space to get the number representing the next state. Each round of assembly sequence conforms to the Markov decision process. By default, the assembly state is transferred n times if the assembly is successful every time. Each action a is selected from the action space A. An assembly step can only appear once. When the state is transferred, the action space should be subtracted from the action selected in the previous step. It is necessary to define the reward function in combination with the contact relationship matrix, prohibited sequence matrix, and priority matrix of the model. Here, we define the reward value for triggering the contact relationship matrix as 4, the reward value for triggering the prohibited sequence planning as -10, and the situation of triggering the priority matrix is ​​determined according to the difference in levels between the front and rear parts. If the latter action is at the same level as the previous action, the reward value is defined as 1. If the latter action is one level lower than the previous action, the reward value is defined as 0.5, the reward value is defined as -3 for two levels lower, and the reward value is defined as -4 for three levels lower.

4. A reinforcement learning hybrid method for assembly sequence planning according to claim 1, characterized in that: In step S3, before the Q-value of an action in a certain state is updated, both the update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm are used to perform a calculation in advance, and then the Q-values ​​obtained by the two are compared, and the one with the larger value is used for this update.

5. A reinforcement learning hybrid method for assembly sequence planning according to claim 4, characterized in that: The Q-learning algorithm is specifically: Initialize the Q function corresponding to the actions in all states (usually zero or a small random value), select an action a according to the current state s and strategy (such as ∈-greedy strategy), execute action a, observe the reward r of the environment feedback and the new state s′ transferred to, and then repeat the above actions until the algorithm converges. The Sarsa algorithm is specifically: Initialize the Q function corresponding to the actions in all states (usually zero or a small random value), select an action a according to the current state s and strategy (such as ∈-greedy strategy), execute action a, observe the reward r of the environment feedback and the new state s′ transferred to. Different from the Q-learning algorithm, at this time, the next action a′ will be selected according to the new state s′ using the same strategy, and the Q function will be updated according to the update formula of the Sarsa algorithm, and then the above actions will be repeated until the algorithm converges.

6. A reinforcement learning hybrid method for assembly sequence planning according to claim 4, characterized in that: The update formula of the Q-learning algorithm and the update formula of the Sarsa algorithm are respectively: Sarsa algorithm update formula: Q(s,a)←Q(s,a)+α[r+γQ(s′,a′)-Q(s,a)](4) Q-learning algorithm update formula: Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′,a′)-Q(s,a)](5) Where: s represents the current state; a represents the action selected in the current state; s′ represents the state after taking action a; a′ represents the next action selected in state s′; α represents the learning rate, which controls the step size of the weight adjustment during each update of the model. The learning rate determines the step size of the model parameters moving towards the optimal solution at each iteration. r represents the reward obtained by taking an action, which depends on the state s and the action a γ represents the discount factor, which is used to balance immediate rewards and future rewards. The discount factor determines whether the algorithm should pay more attention to immediate rewards or future rewards when making decisions. Q(s,a) represents the Q value corresponding to action a in state s; Q(s′,a′) represents the Q value corresponding to taking action a′ in state s′; max a′ Q(s′,a′) represents the maximum Q value of all actions in state s′.

7. A reinforcement learning hybrid method for assembly sequence planning according to claim 4, characterized in that: In step S3, the initial definition of the ε value is not in the range of 0 to 1, but a value greater than 1, and it is stipulated that each time the ε-greedy strategy is used, the ε value will be gradually reduced until the required value is reached.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the reinforcement learning hybrid method for assembly sequence planning as claimed in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the reinforcement learning hybrid method for assembly sequence planning as claimed in any one of claims 1 to 7 is implemented.

Images