Workshop efficient real-time intelligent scheduling method and system based on deep reinforcement learning

Abstract

The invention provides a workshop high-efficiency real-time intelligent scheduling method based on deep reinforcement learning. The workshop high-efficiency real-time intelligent scheduling method comprises the steps of S1, constructing workshop training simulation information; s2, constructing a workshop vehicle parallel system based on the workshop training simulation information, and executing the workshop vehicle parallel system to output workshop snapshot information in real time; and S3, receiving the workshop snapshot information, inputting the workshop snapshot information into a neural network, and outputting a scheduling instruction. Based on the built workshop training simulation information, the workshop vehicle parallel system is built, and the workshop vehicle parallel system is executed to output workshop snapshot information in real time, so that the neural network is trained, the production process is optimized, it is ensured that vehicles move efficiently and orderly, waiting and congestion are reduced, and the overall scheduling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent scheduling, and in particular to a method and system for efficient real-time intelligent scheduling of workshops based on deep reinforcement learning. Background Art

[0002] In workshop scheduling, it is necessary to manually consider multiple attributes such as the vehicle models, planned production ratios, offline time, types (normal vehicles, reserved vehicles, empty vehicles, etc.) of multiple vehicles. It has the characteristics of strong time series dependence, multi-dimensionality, and high complexity of scheduling. It requires manual preparation of relatively complex track rules and timely takeover of scheduling. Track rules refer to: a set of guiding principles in intelligent workshop scheduling, which are formulated by technical personnel based on production needs and workshop conditions. These rules determine the running order, storage location, and scheduling timing of vehicles on the track to optimize the production process, ensure efficient and orderly movement of vehicles, reduce waiting and congestion, and improve overall scheduling efficiency.

[0003] At present, vehicle manufacturers generally use a combination of technicians compiling track operation rules and manual flexible scheduling to dispatch workshop vehicles. However, due to the various attributes of vehicles, manual scheduling is difficult to handle the high-dimensional information of the entire workshop, and requires extremely high energy and concentration from people, which easily leads to fatigue and slackness of workers, resulting in scheduling efficiency not meeting the standard. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a method and system for efficient real-time intelligent scheduling of workshops based on deep reinforcement learning, so as to solve the technical problems in the prior art that due to the numerous attributes of vehicles, manual scheduling is difficult to handle the high-dimensional information of the entire workshop, it requires extremely high energy and concentration from people, and it is easy for workers to become tired and relaxed, resulting in scheduling efficiency not meeting the standards.

[0005] The present invention provides a workshop efficient real-time intelligent scheduling method based on deep reinforcement learning, comprising:

[0006] S1. Construct workshop training simulation information;

[0007] S2, constructing a workshop-vehicle parallel system based on the workshop training simulation information, and executing the workshop-vehicle parallel system to output workshop snapshot information in real time;

[0008] S3. Receive the workshop snapshot information, input the workshop snapshot information into a neural network, and output a scheduling instruction.

[0009] Optionally, constructing workshop training simulation information includes:

[0010] The workshop consists of PBS area entrance, PBS area exit, PBS track, control point, vehicle attributes, total number of training vehicles, vehicle type ratio, approaching prohibited vehicle type, reserved vehicle type and threshold.

[0011] Optionally, the executing the workshop-vehicle parallel system to output workshop snapshot information in real time includes:

[0012] The parallel system simulates the control flow of the vehicle from the entrance to the PBS area to the exit of the PBS area based on the workshop training simulation information, and outputs workshop snapshot information.

[0013] Optionally, the receiving the shop snapshot information, inputting the shop snapshot information into a neural network, and outputting a scheduling instruction includes:

[0014] S301, receiving the workshop snapshot information of the specified interface and converting it into a state variable S t , expressed as:

[0015] S t ={Attr -1-74 , ΔTrimo, ΔRadio}

[0016] Attr n ={type, attr, keepoff}

[0017] Among them, Attr n is the vehicle attribute in the workshop track, Attr -1-74 is the vehicle attribute of all vehicles in the workshop, ΔTrimo is the difference between the number of vehicles stored in the next workshop and the threshold, and ΔRadio is the difference between the proportion of vehicles stored on the day and the set proportion of vehicles;

[0018] S302: The state variable S t Input Actor network, output scheduling instructions, expressed as:

[0019] A t ={STRG 1-7 , Return_RET1, Is_wait}

[0020] Among them, STRG 1-7 Indicates one of the corresponding vehicle tracks, Return_RET1 indicates whether the track result is sent back to RET-1, and Is_wait indicates whether the wait command result is executed.

[0021] The present invention also provides a workshop efficient real-time intelligent scheduling system based on deep reinforcement learning, comprising:

[0022] Information construction module, constructing workshop training simulation information;

[0023] A vehicle parallel module, which constructs a vehicle parallel system in a workshop based on the workshop training simulation information, and executes the vehicle parallel system in a workshop to output workshop snapshot information in real time;

[0024] The scheduling output module receives the workshop snapshot information, inputs the workshop snapshot information into the neural network, and outputs scheduling instructions.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] Based on the constructed workshop training simulation information, a workshop-vehicle parallel system is constructed, and the workshop-vehicle parallel system is executed to output workshop snapshot information in real time, so as to train the neural network, optimize the production process, ensure the efficient and orderly movement of vehicles, reduce waiting and congestion, and improve the overall scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0030] Figure 2 FIG. 4 is a schematic diagram of the PBS area layout of a workshop according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation cases obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The functional units with the same labels in the examples of the present invention have the same and similar structures and functions.

[0032] See also Figure 1 The present invention provides a workshop efficient real-time intelligent scheduling method based on deep reinforcement learning, comprising:

[0033] S1. Construct workshop training simulation information;

[0034] S2, constructing a workshop-vehicle parallel system based on the workshop training simulation information, and executing the workshop-vehicle parallel system to output workshop snapshot information in real time;

[0035] S3. Receive the workshop snapshot information, input the workshop snapshot information into a neural network, and output a scheduling instruction.

[0036] In this embodiment, S1, construct workshop training simulation information. This workshop consists of: MAIN0, MAIN1, MAIN2, MAIN3, MAIN4, MAIN5, MAIN6, STRG1, STRG2, STRG3, STRG4, STRG5, STRG6, STRG7, RET-1, where MAIN0 is the workshop entrance, MAIN6 is the workshop exit, RET-1 is the track for vehicles to return from the sorting exit of the PBS area to the sorting entrance of the PBS area, and MAIN1, MAIN2, MAIN3, MAIN4, MAIN5, STRG1, STRG2, STRG3, STRG4, STRG5, STRG6, STRG7, RET-1 are all tracks;

[0037] Vehicle attributes: vehicle model code; whether it is a reserved vehicle; whether it is an empty vehicle;

[0038] Request control point: The connection between the tracks in this workshop includes two types: many-to-one and one-to-many. The point of intelligent scheduling control is many-to-one, that is, one track is selected from multiple tracks to the specified track;

[0039] Number of training vehicles N: In training mode, the total number of vehicles released from MAIN0 can be set through the UI interface. The default value is 5000. The system will randomly select a starting vehicle from historical data, and then place N vehicles in the PBS area in sequence. The PBS area is a sorting area in the general assembly workshop to participate in training. The number of vehicles can also be set to randomly select the released vehicles.

[0040] The planned production ratio of vehicle models indicates the planned production ratio of vehicle models every day of each month. It needs to be pre-configured on the interface: the setting format is [A:B:C=1:2:2], where A, B, C are vehicle models, the vehicle model serial number is used as the basis for distinguishing vehicle types, and 1:2:2 is the ratio of vehicles expected to be released on that day;

[0041] Prohibited models: The format is set to [A,B]. Since the processing time of this type of model is long, in order to ease the difficulty of downstream processing, they are arranged separately, indicating that adjacent models are prohibited;

[0042] Reserved vehicle model, indicating that the current vehicle model can only be stored in the PBS area: the setting format is [C,D], C, D respectively indicate that the reserved vehicle model can be set or the number of vehicles allowed to be released from the warehouse can be designed, and the PBS area cannot be released by MAIN6;

[0043] Trimo threshold is the threshold set for the Trimo area. The Trimo area is the last buffer area before entering the final assembly. The number of vehicles in this area is directly related to the production rhythm of the assembly line: the Trimo value is the number of vehicles stored in the next PBS area. When the current Trimo value is less than the threshold, MAIN6 must release the vehicle to ensure the normal operation of the downstream PBS area. The default value is 40;

[0044] S2. Construct a PBS area vehicle parallel system based on the PBS area training simulation information, and execute the PBS area vehicle parallel system to output PBS area snapshot information in real time.

[0045] (2) In parallel mode, the information of real vehicles entering the PBS and exiting the PBS in the production environment is synchronized in real time, and the intermediate control process is controlled by AI outputting control signals. The order, time, and proportion of vehicles entering the PBS area will be synchronized with the actual production. The initial synchronization operation will synchronize the track and PBS area status of the parallel mode with the status of the actual production environment as a benchmark.

[0046] See also Figure 2 The following will describe the complete control process of the vehicle from the PBS area entrance MAIN0 to the PBS area exit MAIN6 in parallel mode:

[0047] CH: Control point H, receiving the vehicle from MAIN0 to MAIN1, triggering the operation mechanism when the last parking space in MAIN1 is empty and MAIN0 is not empty; for example, translated as: Step 1, when the vehicle arrives at the entrance of the PBS area, determine whether MAIN1 in the PBS area is the last parking space, if so and MAIN0 is not empty, transport the vehicle from the entrance of the PBS area to MAIN1, and execute step S2 (CI), otherwise no response;

[0048] CI: Control point I, if OUT-1 is in occupied state and OUT-2 value is greater than 9, and the actual number of empty cars in RET3 is greater than the threshold configured on the system, it indicates that the empty cars outside the PBS area are full and need to be sent into the PBS area to prevent congestion, then execute the CH1 command: OUT1->MAIN2; otherwise execute the CH2 command MAIN1->MAIN2 (Out1 and out2 represent the tracks for empty cars to enter and exit the PBS area, respectively);

[0049] CA: Control point A, if the number of vehicles in RET-1 is 17 (limited by on-site production business, the control logic does not only determine whether RET-1 is full, but also other factors are learned and judged by the neural network itself), indicating that RET-1 is full, then execute CA1 command: RET-1->MAIN3; otherwise execute CA2 command MAIN2->MAIN3;

[0050] CB: Control point B, if the vehicle type of MAIN3 is empty and RET-2 is empty, execute the CB1 command: MAIN3->RET-2. RET-2 is the track for empty vehicles in the PBS area to return to OUT2. When the number of vehicles in OUT2 is less than or equal to the set threshold, it means that OUT2 can store excess empty vehicles. If there are empty vehicles in the PBS area, they will be transferred to OUT2 via OUT-1 through the RET-2 track;

[0051] Otherwise, the following principle is executed: extract the track data of STRG1, STRG2, STRG3, STRG4, STRG5, STRG6, and STRG7 in sequence: the attributes of the last vehicle [empty vehicle / normal vehicle / reserved vehicle, vehicle type, and number of vehicles], and match the three attributes of the vehicle in sequence, so that the vehicles with the same attributes are placed on the same track. Execute the command set according to this rule:

[0052] CB2: MAIN3->MAIN4(STRG1); CB3: MAIN3->MAIN4(STRG2); CB4: MAIN3->MAIN4(STRG3); CB5: MAIN3->MAIN4(STRG4); CB6: MAIN3->MAIN4(STRG5); CB7: MAIN3->STRG6; CB8: MAIN3->STRG7;

[0053] Since the MAIN3 track has a buffer track MAIN4 with the STRG1, STRG2, STRG3, STRG4, and STRG5 tracks, vehicles entering STRG1, STRG2, STRG3, STRG4, and STRG5 need to execute more CC commands in sequence;

[0054] CC: Control point C, execute command set: CC1: MAIN4->STRG1; CC2: MAIN4->STRG2; CC3: MAIN4->STRG3; CC4: MAIN4->STRG4; CC5: MAIN4->STRG5;

[0055] CD: Control point D, which is the intelligent control point of the deep reinforcement learning algorithm. The execution command set is:

[0056] CD1:STRG1->MAIN5;

[0057] CD2:STRG2->MAIN5;

[0058] CD3:STRG3->MAIN5;

[0059] CD4:STRG4->MAIN5;

[0060] CD5:STRG5->MAIN5;

[0061] CD6:STRG6->MAIN5;

[0062] CD7:STRG7->MAIN5;

[0063] CD8:STRG1->MAIN5(RET-1);

[0064] CD9:STRG2->MAIN5(RET-1);

[0065] CD10:STRG3->MAIN5(RET-1);

[0066] CD11:STRG4->MAIN5(RET-1);

[0067] CD12: STRG5->MAIN5(RET-1);

[0068] CD13:STRG6->MAIN5(RET-1);

[0069] CD14:STRG7->MAIN5(RET-1);

[0070] CD15: WAIT;

[0071] CE: Control point E. If the command of control point D is one of CD8-CD14, execute CE1:MAIN5->RET-1, otherwise execute CE2:MAIN5->MAIN6;

[0072] CF: control point F, command CF1: MAIN6->TRIM area;

[0073] Among them, the TRIM area is a part of the PBS area. It is a buffer area after sorting and before officially entering the final assembly line. The order of vehicles in this area is fixed and cannot be sorted again. Vehicles in this area are lowered to the production line through elevators in sequence for final assembly. The scheduling strategy of the sorting area must ensure that the number of vehicles in the TRIM area does not fall below a certain threshold, otherwise it may affect the entire production rhythm.

[0074] Finally, this mode will calculate manual scores and intelligent scheduling algorithm scores, and generate daily vehicle sequence information and other log information.

[0075] S3. Receive the workshop snapshot information, input the workshop snapshot information into a neural network, and output a scheduling instruction.

[0076] (3) The intelligent scheduling algorithm obtains the current workshop snapshot through the interface, inputs the state information S into the fully connected neural network structure of the intelligent agent, and outputs the scheduling instructions of the control point D. The pseudo code and calculation formula are as follows:

[0077] S301: Receive the workshop snapshot information through the specified interface and convert it into the following state variable S t :

[0078] S t ={Attr -1-74 , ΔTrimo, ΔRadio}

[0079] Attr n ={type, attr, keepoff}

[0080] In the formula, Attr n The attributes of the vehicles in STRG1, STRG2, STRG3, STRG4, STRG5, STRG6, and STRG7: type [A / B / C / D], attr [empty vehicle / red vehicle (reserved vehicle) / normal vehicle], whether keepoff is approach prohibition [True / False], using digital encoding; Attr -1 is the attribute of the last vehicle to leave the workshop; ΔTrimo is the difference between the current Trimo value and the threshold; ΔRadio is the difference between the proportion of vehicle models released on the day and the set proportion of vehicle models;

[0081] S302: Determine the inference device [graphics card / CPU] of the current device, load the pre-trained PTH model weights of the specified device, and initialize the algorithm parameters, including learning rate 1e-3, discount factor y0.99, entropy regularization coefficient α0.01, soft update coefficient τ0.005, and exploration noise ∈0.1. t Pass in the Actor network and get the network output ACTION:

[0082] A t ={STRG 1-7 , Return_RET1, Is_wait}

[0083] In the formula, STRG 1-7 Corresponding to one of the tracks STRG1, STRG2, STRG3, STRG4, STRG5, STRG6, STRG7, Return_RET1 is whether to return to the RET-1 track. If true, execute one of CD8-CD14, otherwise execute one of CD1-CD7. Is_wait is whether to execute the wait command. If true, execute command CD15;

[0084] S303: After executing the control point D command, the action score is calculated:

[0085]

[0086] Where: r1 is whether approach prohibition occurs, [yes / 0, no / 1];

[0087] r2 is whether to circle, that is, whether Return_RET1 is true, [yes / 0, no / 1];

[0088] r3 is the vehicle model ratio score,

[0089] r4 is the score for consecutive appearances of vehicles of the same type: C i is the number of consecutive identical elements starting from the i-th element in the sequence;

[0090] S304: Experience storage, at time t+1, {S t , A t , R t , S t+1 , Done} is stored in experience pool D;

[0091] S305: When the number of experience pools reaches MIN_NUM, obtain B pieces of data from experience pool D and perform the following training:

[0092] S3051: Using Strategy Optimization Formula Update the Actor network, where J(π) is the strategy P π The performance function, r(s t , a t ) is in state s t Execute action a t The immediate reward obtained, α is the entropy weight parameter used to balance the reward and entropy, H(π(·|s t ) is the state s under the strategy t Entropy under

[0093] S3052: Update formula using value function Update the Q function of the Critic network, where Q(s t , a t Is to perform action a t In status t The soft Q value, V(s t+1 ) is in state s t+1 The soft value function under , γ is the discount factor of future rewards;

[0094] S3053: Update soft value function Reflects the expected return of taking a certain action under the current strategy, where V(s t) Real state s t The soft value function under logπ(a t |s t ) indicates the execution of action a t The logarithmic probability of the strategy reflects the contribution of entropy.

[0095] S3054: Use the soft update coefficient τ to smoothly update the target network to reduce the instability during training. The parameters of the target network are updated to θ target ←τθ+(1-τ)θ target , where θ is the parameter of the current network.

[0096] The present invention constructs a workshop-vehicle parallel system based on the constructed workshop training simulation information, and executes the workshop-vehicle parallel system to output workshop snapshot information in real time, so as to train the neural network, optimize the production process, ensure the efficient and orderly movement of vehicles, reduce waiting and congestion, and improve the overall scheduling efficiency.

[0097] The present invention also provides a workshop efficient real-time intelligent scheduling system based on deep reinforcement learning, comprising:

[0098] Information construction module, constructing workshop training simulation information;

[0099] A vehicle parallel module, which constructs a vehicle parallel system in a workshop based on the workshop training simulation information, and executes the vehicle parallel system in a workshop to output workshop snapshot information in real time;

[0100] The scheduling output module receives the workshop snapshot information, inputs the workshop snapshot information into the neural network, and outputs scheduling instructions.

[0101] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0102] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A workshop efficient real-time intelligent scheduling method based on deep reinforcement learning, characterized in that: include: S1. Construct workshop training simulation information; S2, constructing a workshop-vehicle parallel system based on the workshop training simulation information, and executing the workshop-vehicle parallel system to output workshop snapshot information in real time; S3. Receive the workshop snapshot information, input the workshop snapshot information into a neural network, and output a scheduling instruction.

2. The workshop efficient real-time intelligent scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The construction of workshop training simulation information includes: The workshop consists of PBS area entrance, PBS area exit, PBS track, control point, vehicle attributes, total number of training vehicles, vehicle type ratio, approaching prohibited vehicle type, reserved vehicle type and threshold.

3. The workshop efficient real-time intelligent scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The executing of the workshop-vehicle parallel system to output workshop snapshot information in real time includes: The parallel system simulates the control flow of the vehicle from the entrance to the PBS area to the exit of the PBS area based on the workshop training simulation information, and outputs workshop snapshot information.

4. The workshop efficient real-time intelligent scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The receiving the shop snapshot information, inputting the shop snapshot information into a neural network, and outputting a scheduling instruction comprises: S301, receiving the workshop snapshot information of the specified interface and converting it into a state variable S t , expressed as: S t ={Attr -1-74 ,ΔTrimo,ΔRadio} Attr n ={type,attr,keepoff} Among them, Attr n is the vehicle attribute in the workshop track, Attr -1-74 is the vehicle attribute of all vehicles in the workshop, ΔTrimo is the difference between the number of vehicles stored in the current trimo area and the set threshold, ΔRadio is the difference between the proportion of vehicle models released on the day and the set proportion of vehicle models; S302: The state variable S t Input Actor network, output scheduling instructions, expressed as: A t ={STRG 1-7 ,Return_RET1,Is_wait} Among them, STRG 1-7 Indicates one of the corresponding vehicle tracks, Return_RET1 indicates whether the track result is sent back to RET-1, and Is_wait indicates whether the wait command result is executed.

5. A workshop efficient real-time intelligent scheduling system based on deep reinforcement learning, characterized in that: include: Information construction module, constructing workshop training simulation information; A vehicle parallel module, which constructs a vehicle parallel system in a workshop based on the workshop training simulation information, and executes the vehicle parallel system in a workshop to output workshop snapshot information in real time; The scheduling output module receives the workshop snapshot information, inputs the workshop snapshot information into the neural network, and outputs scheduling instructions.

Images