Generation method of PBS area scheduling system, electronic equipment and readable storage medium
Through the construction and training of deep reinforcement learning models, a more efficient and intelligent PBS area scheduling system is generated, which solves the problems of insufficient flexibility, complexity and optimization capabilities of existing scheduling algorithms, and achieves more efficient production process management and optimization.
Patent Information
- Application Number
- CN202411404417.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-02-28
AI Technical Summary
The existing PBS area scheduling algorithm lacks flexibility, is difficult to handle complex scheduling, is slow to respond, has limited optimization capabilities, poor scalability and high maintenance costs, and lacks automation and intelligence.
The deep reinforcement learning model is adopted, including the policy network and the value network. By constructing and training the model, historical scheduling data are obtained, the actual and expected values of the reward function are determined, and the parameters of the policy network are adjusted to generate a more efficient and intelligent PBS area scheduling system.
It realizes more efficient production process management and optimization, improves production efficiency and resource utilization, reduces costs, and enhances the overall performance and flexibility of the system.
Smart Images
Figure CN119539327A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to scheduling technology, and more specifically, to a generation method, electronic device and readable storage medium of a PBS area scheduling system. Background Art
[0002] PBS (Painted Body Store) is an indispensable part of modern automobile manufacturing. It not only improves the efficiency and flexibility of the production line, but also plays an important role in quality control and inventory management. Through PBS, automobile manufacturers can better respond to changes in market demand. The business scenarios of PBS are as follows: the car body is sprayed in the paint shop, including multiple spraying steps such as primer, topcoat and clearcoat, and is ready to enter the PBS area after baking and cooling; the car bodies stored in the PBS area are arranged according to specific rules for easy management and retrieval; according to the production plan, the car bodies in the PBS area are shipped out on demand and enter the general assembly workshop, and then various parts are assembled to finally complete the manufacturing of the whole vehicle.
[0003] Most of the existing PBS zone scheduling algorithms are based on rules developed by artificial experience. Although the PBS zone scheduling algorithms based on artificial rules can work effectively in some cases, their shortcomings and disadvantages become obvious as the complexity and scale of the manufacturing process increase. The main manifestations are as follows: lack of flexibility and inability to handle complex scheduling. Scheduling algorithms based on manual rules usually rely on a predefined set of rules and logic, which are difficult to quickly adapt to changes in the production environment. For complex scheduling, multiple variables and constraints are involved. Scheduling algorithms based on manual rules find it difficult to fully consider these factors and perform complex multi-objective optimization; slow response. Scheduling algorithms based on manual rules react slowly to emergencies (such as equipment failures, supply chain disruptions, etc.); limited optimization capabilities. Scheduling algorithms based on manual rules are often based on experience and intuition and lack data-driven optimization capabilities; poor scalability and high maintenance costs. With the expansion of production lines and changes and complexity of production needs, scheduling algorithms based on manual rules need to be continuously expanded, updated and maintained. The expansion is difficult and the maintenance cost is high, which may lead to the complexity and redundancy of manual rules and affect the overall performance of the system; insufficient data utilization. Scheduling algorithms based on manual rules usually lack full utilization of historical data and real-time data, and cannot extract valuable patterns and insights from them to optimize the scheduling process; lack of automation and intelligence. Scheduling algorithms based on manual rules lack automation and intelligence and cannot learn and improve autonomously.
[0004] Therefore, it is necessary to provide a new PBS area scheduling technology solution to achieve more efficient and intelligent scheduling.
[0005] Public Content
[0006] One object of the present disclosure is to provide a new technical solution for a method of generating a PBS zone scheduling system.
[0007] According to a first aspect of the present disclosure, a method for generating a PBS zone scheduling system is provided, comprising:
[0008] Constructing a deep reinforcement learning model, wherein the deep reinforcement learning model includes a policy network and a value network;
[0009] Acquiring historical scheduling data, the historical scheduling data including track status information of the PBS area, scheduling action information of the vehicle body before entering the assembly area, and vehicle body sequence information entering the assembly area;
[0010] Determine an actual reward value of a reward function corresponding to the historical scheduling data, and determine an expected reward value corresponding to the historical scheduling data using the value network;
[0011] Determine a loss function value according to the actual reward value and the expected reward value;
[0012] When the loss function value exceeds a preset threshold, the parameters of the policy network are adjusted to adjust the vehicle body scheduling strategy until the loss function value is less than or equal to the preset threshold, thereby completing the training of the deep reinforcement learning model to generate a PBS zone scheduling system determined based on the deep reinforcement learning model.
[0013] Optionally, determining an actual reward value of a reward function corresponding to the historical scheduling data includes:
[0014] Determining a first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area;
[0015] Determine a second actual reward value according to the manually formulated dispatch rule information;
[0016] The actual reward value is determined according to the first actual reward value and the second actual reward value.
[0017] Optionally, determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area includes:
[0018] Determine whether an infeasible scheduling action occurs to the vehicle body during the vehicle body scheduling process according to the track state information of the PBS area and the scheduling action information of the vehicle body before entering the assembly area, and obtain a first actual reward value score;
[0019] Obtaining a second actual reward value score according to whether the vehicle body sequence information entering the assembly area meets a preset sequence requirement;
[0020] The first actual reward value is determined according to the first actual reward value and the second actual reward value.
[0021] Optionally, the preset sequence requirement is that the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds a first preset degree value, the vehicle bodies whose assembly levels exceed the preset level cannot be continuously shipped out of the warehouse, and the matching degree value between the order of the vehicle bodies entering the assembly area and the order of the vehicle order time exceeds a second preset degree value.
[0022] The obtaining of a second actual reward value score according to whether the vehicle body sequence information entering the assembly area meets the preset sequence requirement includes:
[0023] Determine whether the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds a first preset degree value, and / or determine whether the vehicle bodies whose assembly levels exceed the preset level are continuously released from the warehouse, and / or determine whether the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order times exceeds a second preset degree value, and obtain the second actual reward value score.
[0024] Optionally, the manually formulated dispatch rule information is at least one of the following: the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, the frame of the dispatched vehicle body cannot enter the assembly area, and the vehicle body corresponding to a pre-calibrated vehicle body model cannot enter the assembly area, wherein:
[0025] The determining of the second actual reward value according to the manually formulated dispatch rule information includes:
[0026] Determine whether the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, and / or determine whether the frame of the scheduled vehicle body enters the assembly area, and / or determine whether the vehicle body corresponding to a pre-calibrated vehicle body model enters the assembly area, to obtain the second actual reward value.
[0027] Optionally, the historical scheduling data also includes the number of existing frames in the frame area during the vehicle body scheduling process, wherein:
[0028] Determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area includes:
[0029] The first actual reward value is determined according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, the vehicle body sequence information entering the assembly area, and the number of existing frames in the frame area during the vehicle body scheduling process.
[0030] Optionally, using the value network to determine the expected reward value corresponding to the historical scheduling data includes:
[0031] According to the track status information and the production plan information, the value network is used to determine the expected reward value corresponding to the historical scheduling data.
[0032] Optionally, adjusting the parameters of the strategy network to adjust the vehicle body scheduling strategy includes:
[0033] The PPO algorithm is used to adjust the parameters of the strategy network to adjust the vehicle body scheduling strategy.
[0034] According to a second aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute any one of the methods described in the first aspect of the present disclosure.
[0035] According to a third aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of the first aspects is implemented.
[0036] The method for generating a PBS zone scheduling system provided by an embodiment of the present invention and the generated PBS zone scheduling system can more effectively manage and optimize the production process, improve production efficiency and resource utilization, reduce costs, and enhance the overall performance and flexibility of the system.
[0037] Features and advantages of the embodiments of the present specification will become apparent from the following detailed description of exemplary embodiments of the present specification with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the specification and, together with the description, serve to explain the principles of the embodiments of the specification.
[0039] Figure 1 4 is a processing flow chart of a method for generating a PBS zone scheduling system according to an embodiment of the present invention.
[0040] Figure 2 is a schematic diagram of a PBS scheduling area according to an embodiment of the present invention.
[0041] Figure 3 FIG. 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0042] Various exemplary embodiments of the present specification will now be described in detail with reference to the accompanying drawings.
[0043] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the embodiments of the present specification and its application or uses.
[0044] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0045] In one embodiment of the present invention, a method for generating a PBS zone scheduling system is provided. Figure 1 As shown, the method for generating the PBS zone scheduling system of this embodiment may include the following steps S110 to S150.
[0046] Step S110, constructing a deep reinforcement learning model, the deep reinforcement learning model includes a policy network and a value network.
[0047] Before training the deep reinforcement learning model, initialize the parameters of the policy network and the value network.
[0048] Step S120, obtaining historical scheduling data, the historical scheduling data including track status information of the PBS area, scheduling action information of the vehicle body before entering the assembly area, and vehicle body sequence information entering the assembly area.
[0049] The historical scheduling data is used as training data for the deep reinforcement learning model, including multiple sets of training samples. Each set of training samples can be historical scheduling data in units of days. Each set of training samples includes track status information in the PBS area, scheduling action information of the vehicle body before entering the assembly area, and vehicle body sequence information entering the assembly area.
[0050] Step S130, determining the actual reward value of the reward function corresponding to the historical scheduling data, and determining the expected reward value corresponding to the historical scheduling data using the value network.
[0051] The actual reward value of the reward function corresponding to the historical scheduling data is the reward value generated during the vehicle body scheduling process.
[0052] Step S140, determining a loss function value according to the actual reward value and the expected reward value.
[0053] According to the actual reward value and the expected reward value, the difference between the actual reward value and the expected reward value is calculated, and the difference is used as the loss function value.
[0054] Step S150, when the loss function value exceeds the preset threshold, adjust the parameters of the strategy network to adjust the vehicle body scheduling strategy until the loss function value is less than or equal to the preset threshold, completing the training of the deep reinforcement learning model to generate a PBS zone scheduling system determined based on the deep reinforcement learning model.
[0055] Through training, the policy network will learn actions that adapt to different environments and task requirements to maximize the reward value.
[0056] The method for generating a PBS zone scheduling system provided in an embodiment of the present invention and the generated PBS zone scheduling system can more effectively manage and optimize the production process, improve production efficiency and resource utilization, reduce costs, and enhance the overall performance and flexibility of the system.
[0057] In one embodiment, determining the actual reward value of the reward function corresponding to the historical scheduling data specifically includes: determining a first actual reward value based on the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area; determining a second actual reward value based on the manually formulated scheduling rule information; determining the actual reward value based on the first actual reward value and the second actual reward value.
[0058] The first actual reward value is the reward value of environmental feedback, including the reward value obtained by judging whether the scheduling action is feasible in the environment and whether to encourage it, and the reward value obtained by the sequence information of the car body entering the assembly area. The scheduling action is to move the car body from one point to another. The car body is placed on the frame, and the scheduling of the car body is completed by moving the frame. The real-time position information of the car body can be directly obtained. The scheduling action of the car body can be determined based on the changes in the two adjacent real-time positions of the car body.
[0059] Figure 2 The schematic diagram of the PBS dispatching area of one embodiment of the present invention is shown in FIG. The vehicle bodies in the PBS dispatching area come from the painting area, and the vehicle bodies in the PBS dispatching area are finally shipped out to the assembly area.
[0060] See also Figure 2 , the black lines represent the tracks, and the arrows indicate the feasible scheduling directions of the current track body. Each square represents the placement of the body. DED, DGA and DBD are all body models, and E represents the frame.
[0061] The second actual reward value is a reward value based on feedback from manually formulated scheduling rules.
[0062] In this embodiment, artificial rules are embedded in the reward function. During the training process, it is guided not only by reward signals from the environment but also by the prior knowledge provided by the artificial rules, which helps to accelerate the training process and ensure that unsafe or inappropriate actions are not performed before the appropriate strategy is learned.
[0063] In one embodiment, the actual reward value is determined based on the following calculation:
[0064] R 实 =R1+α*R2
[0065] Among them, R 实 is the actual reward value, R1 is the first actual reward value, R2 is the second actual reward value, and α is the weight parameter. α can be set according to requirements.
[0066] In one embodiment, the first actual reward value can be determined according to the following steps: determine whether the vehicle body has an infeasible scheduling action during the vehicle body scheduling process based on the track status information of the PBS area and the scheduling action information of the vehicle body before entering the assembly area, and obtain a first actual reward value score; obtain a second actual reward value score based on whether the vehicle body sequence information entering the assembly area meets the preset sequence requirements; determine the first actual reward value based on the first actual reward value score and the second actual reward value score.
[0067] In one embodiment, the first actual reward value score and the second actual reward value score are weighted and summed to obtain the first actual reward value. The weight coefficients of the first actual reward value score and the second actual reward value score are pre-stored values and can be set according to requirements.
[0068] In one embodiment, based on the track status information of the PBS area and the scheduling action information of the vehicle body before entering the assembly area, it is determined whether the vehicle body scheduling process before entering the assembly area enters a track with the track status information being closed, and whether the vehicle body scheduling process before entering the assembly area fails to move the vehicle body to the next point, and a first actual reward value score is obtained.
[0069] Before entering the assembly area, the reward value corresponding to the track where the vehicle body scheduling process does not enter the track status information as closed is higher than the reward value corresponding to the track where the vehicle body scheduling process enters the track status information as closed. Before entering the assembly area, the reward value corresponding to moving the vehicle body to the next point in sequence during the vehicle body scheduling process is higher than the reward value corresponding to not moving the vehicle body to the next point during the vehicle body scheduling process.
[0070] The reward value corresponding to the track whose status information is closed but the track entry status information does not appear during the vehicle body scheduling process is high, the reward value corresponding to the track whose status information is closed while the vehicle body is entering the track during the vehicle body scheduling process, the reward value corresponding to moving the vehicle body to the next point in sequence during the vehicle body scheduling process, and the reward value corresponding to not moving the vehicle body to the next point during the vehicle body scheduling process can be set according to needs.
[0071] The weight coefficients of whether the vehicle body scheduling process before entering the assembly area has a reward value corresponding to whether the track status information is closed and whether the vehicle body scheduling process before entering the assembly area has a reward value corresponding to whether the vehicle body is not moved to the next point. The weight coefficients of the two can be set according to needs.
[0072] In one embodiment, the preset sequence requirements are that at least one of the matching degree between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds a first preset degree value, vehicles whose assembly levels exceed the preset level cannot be shipped out continuously, and the matching degree between the sequence of vehicles entering the assembly area and the sequence of vehicle order times exceeds a second preset degree value.
[0073] The preset body sequence is the arrangement order of different models of bodies determined according to the assembly requirements. The assembly requirements may be assembly efficiency. For example, the current production plan includes three models A, B, and C, and the preset body sequence is AABC. The body sequence entering the assembly area is matched with AABC to obtain a matching degree value, and then it is determined whether the matching degree value exceeds the first preset degree value.
[0074] The assembly level is determined according to the complexity of assembly. The higher the assembly complexity, the higher the assembly level. The assembly complexity of the vehicle body that exceeds the preset level is higher. In order to facilitate the assembly operation of the assembly workers in the assembly area, the vehicle body that exceeds the preset level cannot be continuously shipped out of the warehouse, that is, it cannot continuously enter the assembly area.
[0075] The earlier the vehicle order time is, the earlier the car bodies corresponding to the vehicle order time will enter the assembly area.
[0076] The second actual reward value score can be determined according to the following steps: determining whether the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds the first preset degree value, and / or determining whether the vehicles with assembly levels exceeding the preset level are continuously shipped out, and / or determining whether the matching degree value between the sequence of vehicles entering the assembly area and the sequence of vehicle order time exceeds the second preset degree value, to obtain the second actual reward value score.
[0077] The reward value corresponding to the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeding the first preset degree value is higher than the reward value corresponding to the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence not exceeding the first preset degree value. The reward value corresponding to the vehicle bodies whose assembly level exceeds the preset level but are not continuously shipped out is higher than the reward value corresponding to the vehicle bodies whose assembly level exceeds the preset level continuously shipped out. The reward value corresponding to the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order time exceeding the second preset degree value is higher than the reward value corresponding to the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order time not exceeding the second preset degree value.
[0078] The reward value corresponding to the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeding the first preset degree value, the reward value corresponding to the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence not exceeding the first preset degree value, the reward value corresponding to the vehicle bodies whose assembly levels exceed the preset level not being continuously shipped out of the warehouse, the reward value corresponding to the vehicle bodies whose assembly levels exceed the preset level being continuously shipped out of the warehouse, the reward value corresponding to the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order times exceeding the second preset degree value, the reward value corresponding to the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order times exceeding the second preset degree value, and the reward value corresponding to the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order times not exceeding the second preset degree value can be set according to needs.
[0079] The weight coefficients of whether the matching degree between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds the reward value corresponding to the first preset degree value, whether the vehicle bodies whose assembly levels exceed the preset levels are continuously shipped out of the warehouse, and whether the matching degree between the order of vehicle bodies entering the assembly area and the order of vehicle order times exceeds the reward value corresponding to the second preset degree value can be set according to needs.
[0080] In one embodiment, the manually formulated dispatch rule information is at least one of the following: the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, the frame of the dispatched vehicle body cannot enter the assembly area, and the vehicle body corresponding to a pre-calibrated vehicle body model cannot enter the assembly area.
[0081] The pre-calibrated vehicle body model is pre-stored information and can be directly obtained. When the vehicle body enters the PBS area, the vehicle body model can also be directly obtained.
[0082] In one embodiment, the second actual reward value is determined according to the following steps: determining whether the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, and / or determining whether the frame of the scheduled vehicle body enters the assembly area, and / or determining whether the vehicle body corresponding to a pre-calibrated vehicle body model enters the assembly area to obtain the second actual reward value.
[0083] The reward value corresponding to the number of existing bodies in the assembly area being greater than or equal to the first preset number is higher than the reward value corresponding to the number of existing bodies in the assembly area being less than the first preset number. The reward value corresponding to the frame of the dispatched body not entering the assembly area is higher than the reward value corresponding to the frame of the dispatched body entering the assembly area. The reward value corresponding to the body corresponding to the pre-calibrated body model not entering the assembly area is higher than the reward value corresponding to the frame of the dispatched body entering the assembly area.
[0084] The reward value corresponding to the number of existing vehicle bodies in the assembly area is greater than or equal to the first preset number, the reward value corresponding to the number of existing vehicle bodies in the assembly area is less than the first preset number, the reward value corresponding to the frame of the dispatched vehicle body not entering the assembly area, the reward value corresponding to the frame of the dispatched vehicle body entering the assembly area, the reward value corresponding to the vehicle body corresponding to the pre-calibrated vehicle body model not entering the assembly area, and the reward value corresponding to the frame of the dispatched vehicle body entering the assembly area can be set according to needs.
[0085] In one embodiment, the historical scheduling data also includes the number of existing frames in the frame area during the body scheduling process. Determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the body before entering the assembly area, and the body sequence information entering the assembly area specifically includes: determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the body before entering the assembly area, the body sequence information entering the assembly area, and the number of existing frames in the frame area during the body scheduling process.
[0086] Specifically, according to the track status information of the PBS area and the scheduling action information of the vehicle body before entering the assembly area, it is determined whether an infeasible scheduling action occurs to the vehicle body during the vehicle body scheduling process, and a first actual reward value score is obtained; according to whether the vehicle body sequence information entering the assembly area meets the preset sequence requirements, a second actual reward value score is obtained; according to whether the number of existing frames in the frame area during the vehicle body scheduling process is less than the second preset number, a third actual reward value score is obtained; according to the first actual reward value score, the second actual reward value score and the third actual reward value score, the first actual reward value is determined.
[0087] The reward value corresponding to the number of existing frames in the frame area during the vehicle body scheduling process being less than the second preset number is higher than the reward value corresponding to the number of existing frames in the frame area during the vehicle body scheduling process being greater than or equal to the second preset number.
[0088] In one embodiment, the first actual reward value score, the second actual reward value score and the third actual reward value score are weighted and summed to obtain the first actual reward value. The weight coefficients of the first actual reward value score, the second actual reward value score and the third actual reward value score are pre-stored values and can be set according to needs.
[0089] In one embodiment, the expected reward value corresponding to the historical scheduling data is determined using a value network based on the track status information and the production plan information.
[0090] The production plan information includes the preset sequence information of the car bodies entering the assembly area and the vehicle order time.
[0091] In one embodiment, the expected reward value corresponding to the scheduling data is determined using a value network based on track status information, production plan information, the minimum number of existing vehicle bodies allowed in the assembly area, and the maximum number of temporary storage allowed in the temporary storage area.
[0092] In one embodiment, adjusting the parameters of the policy network to adjust the vehicle body scheduling strategy includes: adjusting the parameters of the policy network using a PPO algorithm to adjust the vehicle body scheduling strategy.
[0093] PPO (Proximal Policy Optimization) is a reinforcement learning algorithm for policy optimization. It is a policy gradient-based method that focuses on optimizing policies to maximize cumulative rewards.
[0094] This embodiment adopts the PPO algorithm to optimize the strategy through repeated trial and error, gradually adapting to the complex learning environment, which can improve learning efficiency, reduce costs, and enhance the overall performance and flexibility of the PBS area scheduling system.
[0095] An embodiment of the present invention provides an electronic device, such as Figure 3 The electronic device 300 includes a memory 320 and a processor 310. The memory 320 stores a computer program, and the computer program is used to control the processor 310 to operate so as to execute the method in any of the above embodiments.
[0096] An embodiment of the present invention provides a non-volatile computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method in any of the above embodiments is performed.
[0097] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the device embodiment, its related parts can be referred to the partial description of the method embodiment.
[0098] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0099] The embodiments of the present specification may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium carrying computer instructions for causing a processor to implement various aspects of the embodiments of the present specification.
[0100] A computer-readable storage medium may be a tangible device that can hold and store computer instructions used by a computer instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which computer instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0101] The computer instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network layer, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network layer may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network layer adapter card or network layer interface in each computing / processing device receives the computer instructions from the network layer and forwards the computer instructions for storage in a computer-readable storage medium in each computing / processing device.
[0102] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to the multiple embodiments of this specification. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a computer instruction, and a part of a module, a program segment or a computer instruction contains one or more executable computer instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0103] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for generating a PBS zone scheduling system, characterized in that: include: Constructing a deep reinforcement learning model, wherein the deep reinforcement learning model includes a policy network and a value network; Acquiring historical scheduling data, the historical scheduling data including track status information of the PBS area, scheduling action information of the vehicle body before entering the assembly area, and vehicle body sequence information entering the assembly area; Determining an actual reward value of a reward function corresponding to the historical scheduling data, and determining an expected reward value corresponding to the historical scheduling data using the value network; Determine a loss function value according to the actual reward value and the expected reward value; When the loss function value exceeds a preset threshold, the parameters of the policy network are adjusted to adjust the vehicle body scheduling strategy until the loss function value is less than or equal to the preset threshold, thereby completing the training of the deep reinforcement learning model to generate a PBS zone scheduling system determined based on the deep reinforcement learning model.
2. The method according to claim 1, characterized in that The determining of the actual reward value of the reward function corresponding to the historical scheduling data includes: Determining a first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area; Determine a second actual reward value according to the manually formulated dispatch rule information; The actual reward value is determined according to the first actual reward value and the second actual reward value.
3. The method according to claim 2, characterized in that Determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area includes: Determine whether an infeasible scheduling action occurs to the vehicle body during the vehicle body scheduling process according to the track state information of the PBS area and the scheduling action information of the vehicle body before entering the assembly area, and obtain a first actual reward value score; Obtaining a second actual reward value score according to whether the vehicle body sequence information entering the assembly area meets a preset sequence requirement; The first actual reward value is determined according to the first actual reward value score and the second actual reward value score.
4. The method according to claim 3, characterized in that The preset sequence requirement is that the matching degree between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds a first preset degree value, the vehicles whose assembly level exceeds the preset level cannot be continuously shipped out, and the matching degree between the order of vehicles entering the assembly area and the order of vehicle order time exceeds a second preset degree value. The obtaining of a second actual reward value score according to whether the vehicle body sequence information entering the assembly area meets the preset sequence requirement includes: Determine whether the matching degree value between the vehicle body sequence entering the assembly area and the preset vehicle body sequence exceeds a first preset degree value, and / or determine whether the vehicle bodies whose assembly levels exceed the preset level are continuously released from the warehouse, and / or determine whether the matching degree value between the sequence of vehicle bodies entering the assembly area and the sequence of vehicle order times exceeds a second preset degree value, and obtain the second actual reward value score.
5. The method according to claim 2, characterized in that: The manually formulated dispatch rule information is at least one of the following: the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, the frame of the dispatched vehicle body cannot enter the assembly area, and the vehicle body corresponding to the pre-calibrated vehicle body model cannot enter the assembly area, wherein: The determining of the second actual reward value according to the manually formulated dispatch rule information includes: Determine whether the number of existing vehicle bodies in the assembly area is greater than or equal to a first preset number, and / or determine whether the frame of the scheduled vehicle body enters the assembly area, and / or determine whether the vehicle body corresponding to a pre-calibrated vehicle body model enters the assembly area, to obtain the second actual reward value.
6. The method according to claim 2, characterized in that The historical scheduling data also includes the number of existing frames in the frame area during the body scheduling process, wherein: Determining the first actual reward value according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, and the vehicle body sequence information entering the assembly area includes: The first actual reward value is determined according to the track state information of the PBS area, the scheduling action information of the vehicle body before entering the assembly area, the vehicle body sequence information entering the assembly area, and the number of existing frames in the frame area during the vehicle body scheduling process.
7. The method according to claim 1, characterized in that The expected reward value corresponding to the historical scheduling data is determined by using the value network, including: According to the track status information and the production plan information, the value network is used to determine the expected reward value corresponding to the historical scheduling data.
8. The method according to any one of claims 1 to 7, characterized in that: The adjusting the parameters of the strategy network to adjust the vehicle body scheduling strategy includes: The PPO algorithm is used to adjust the parameters of the strategy network to adjust the vehicle body scheduling strategy.
9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is used to control the processor to operate so as to execute the method according to any one of claims 1 to 8.
10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.