Intelligent optimization methods, devices, equipment, storage media, and program products for oven parameters of film wrapping machines.
By combining intelligent optimization methods with target feature extraction and strategy networks, the oven parameters of the film wrapping machine are automatically adjusted, solving the problem of low efficiency in manual debugging and achieving rapid adaptation to various working conditions and improved efficiency.
Patent Information
- Application Number
- CN202611242291.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-10
AI Technical Summary
In existing technologies, the adjustment of oven parameters in film wrapping machines relies on manual experience, which requires readjustment every time product specifications are changed, affecting production efficiency and making it impossible to adapt to the diversity of film materials, bottle shapes and production line speeds.
An intelligent optimization method is adopted, which combines a target feature extraction network and a target strategy network with a dual evaluation network to automatically optimize the oven parameters of the film wrapping machine, achieve an adaptive balance between shrinkage effect and yield, and output optimized control parameters.
It enables rapid response to different film materials, bottle shapes, and production line speeds, reducing scrap rates and energy consumption, and improving production line efficiency.
Smart Images

Figure CN123067129A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of oven parameter adjustment technology for film wrapping machines, and in particular to an intelligent optimization method, apparatus, equipment, storage medium, and program product for oven parameters of film wrapping machines. Background Technology
[0002] With the continuous development of automated packaging technology, heat shrink film packaging machines (i.e., film wrapping machines) have been widely used in the bundled packaging of bottled products in industries such as beverages, food, and daily chemicals. The film wrapping machine heats the heat shrink film placed on the outside of the bottle bundle using an oven, causing the film to shrink and tightly adhere to the surface of the bottle bundle, thus achieving bundled packaging. In actual production, multiple process parameters, such as the oven's temperature field distribution, hot air circulation volume, conveyor chain speed, and cooling air volume, jointly determine the film's shrinkage effect and the packaging yield.
[0003] In existing technologies, the parameter adjustment of the oven in a film wrapping machine is usually done manually based on experience. Specifically, the operator manually sets control parameters such as oven temperature, duct fan opening, and conveyor belt speed based on operating conditions such as film material and thickness, bottle size, and production line speed, and repeatedly adjusts them during trial production based on shrinkage effect and yield.
[0004] However, due to the diversity of input conditions such as film material, thickness, shrinkage rate, bottle shape, and packaging size, the optimal shrinkage temperature varies significantly for different film materials. Therefore, different bottle types have different requirements for heat field distribution. In existing technologies, manual readjustment is required every time product specifications are changed, severely impacting production line efficiency. Summary of the Invention
[0005] Therefore, it is necessary to provide a method, apparatus, equipment, storage medium, and program product for intelligent optimization of oven parameters in a film wrapping machine to address the aforementioned technical problems.
[0006] In a first aspect, this application provides a method for intelligent optimization of oven parameters in a film wrapping machine, including:
[0007] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0008] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0009] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0010] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0011] In one embodiment, the current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector, including:
[0012] Obtain the optimized control parameters and historical performance data of the target policy network output in the previous round;
[0013] The numerical parameters in the current physical parameters are normalized to obtain the first input parameter, and the non-numerical parameters in the current physical parameters are embedded and encoded to obtain the second input parameter;
[0014] The first input parameter, the second input parameter, the optimized control parameter output from the previous round, and the historical performance data are concatenated to obtain the current state vector. The current state vector is then input into the target feature extraction network to obtain the shared feature vector.
[0015] In one embodiment, the training process of the target feature extraction network and the target policy network includes:
[0016] Acquire historical experimental data; historical experimental data includes multiple sets of physical parameter samples and control parameter action sample data;
[0017] The sample state vector is determined based on historical experimental data, and then input into the basic feature extraction network to obtain the basic state vector. The basic state vector is then input into the basic policy network to obtain the basic control parameters. Finally, the basic state vector is substituted into the first objective function and the second objective function of the dual evaluation network to obtain the first state parameter and the second state parameter.
[0018] The basic control parameters are input into the film wrapping machine oven to obtain the initial reward data and the sample state vector at the next moment; the initial reward data includes the first reward data for measuring the shrinkage effect and the second reward data for measuring the yield rate;
[0019] The sample state vector, basic control parameters, initial reward data, and the sample state vector at the next moment are stored as a trajectory data in the experience queue.
[0020] When the number of trajectories in the experience queue reaches a preset threshold, for each trajectory data in the experience queue, the first advantage function value and the second advantage function value are calculated respectively based on each initial reward data, each first state parameter and each second state parameter;
[0021] The basic policy network, dual evaluation network, and basic feature extraction network are updated based on the values of the first and second advantage functions until the reward data, shrinkage effect, and yield rate all meet the expected targets, thus obtaining the target feature extraction network and target policy network.
[0022] In one embodiment, the process of updating the basic policy network includes:
[0023] The first policy gradient and the second policy gradient are calculated based on the values of each first advantage function and each second advantage function, respectively.
[0024] When there is a directional conflict between the first policy gradient and the second policy gradient, the first policy gradient and the second policy gradient are projected respectively based on the projection conflict gradient method.
[0025] The base policy network is updated based on the first and second policy gradients after projection processing.
[0026] In one embodiment, the process of updating the dual-evaluation network includes:
[0027] The first value loss value and the second value loss value are obtained based on each first advantage function value and each second advantage function value, respectively.
[0028] The first evaluation gradient and the second evaluation gradient are determined based on the first value loss value and the second value loss value, respectively, and the dual evaluation network is updated based on the first evaluation gradient and the second evaluation gradient.
[0029] In one embodiment, the process of updating the basic feature extraction network includes:
[0030] Based on the first and second policy gradients after projection processing, the policy gradient during the update of the basic policy network is determined.
[0031] Based on the policy gradient, the first evaluation gradient, and the second evaluation gradient, the feature extraction gradient is determined, and the basic feature extraction network is updated according to the feature extraction gradient.
[0032] Secondly, this application also provides an intelligent optimization device for the parameters of a film wrapping machine oven, comprising:
[0033] The acquisition module is used to acquire the current physical parameters of the film wrapping machine oven to be optimized;
[0034] The extraction module is used to input the current physical parameters into a pre-trained target feature extraction network to obtain a shared feature vector;
[0035] The strategy module is used to input the shared feature vector into the pre-trained target strategy network to obtain the optimized control parameters of the membrane packaging oven; wherein, the target feature extraction network and the target strategy network are trained using a dual evaluation network; the dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield target.
[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0037] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0038] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0039] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0040] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0042] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0043] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0044] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0045] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0046] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0047] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0048] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0049] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0050] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0051] The aforementioned intelligent optimization method, device, equipment, storage medium, and program product for the parameters of the film packaging machine oven acquires the current physical parameters of the oven to be optimized; inputs these parameters into a pre-trained target feature extraction network to obtain a shared feature vector; and inputs the shared feature vector into a pre-trained target strategy network to obtain the optimized control parameters of the oven. The target feature extraction network and target strategy network are trained using a dual evaluation network, which includes a first objective function corresponding to the shrinkage effect objective and a second objective function corresponding to the yield rate objective. This method uses the dual evaluation network to evaluate the value of both the shrinkage effect and yield rate objectives and jointly optimizes the strategy network, achieving an adaptive balance between shrinkage effect and yield rate. Simultaneously, by inputting the current physical parameters into the strategy network to output optimized control parameters, it achieves rapid response to various operating conditions such as different film materials, bottle types, packaging sizes, and production line speeds, eliminating the need for repeated manual adjustments. This not only reduces scrap rate and energy consumption but also further improves the production line efficiency. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is an application environment diagram of an intelligent optimization method for oven parameters of a film wrapping machine in one embodiment;
[0054] Figure 2 This is one of the flowcharts illustrating an intelligent optimization method for oven parameters in a film wrapping machine, as described in one embodiment.
[0055] Figure 3 This is a second flowchart illustrating an intelligent optimization method for oven parameters in a film wrapping machine, as described in one embodiment.
[0056] Figure 4 This is the third flowchart illustrating the intelligent optimization method for oven parameters of a film wrapping machine in one embodiment;
[0057] Figure 5 This is a schematic diagram of the policy network structure in one embodiment;
[0058] Figure 6 This is a schematic diagram of a dual-evaluation network structure in one embodiment;
[0059] Figure 7 This is a flowchart illustrating the intelligent optimization method for oven parameters of a film wrapping machine in one embodiment;
[0060] Figure 8 This is the fifth flowchart illustrating an intelligent optimization method for oven parameters in a film wrapping machine, as described in one embodiment.
[0061] Figure 9 This is a flowchart of the intelligent optimization method for oven parameters of a film wrapping machine in one embodiment, shown in Figure 6.
[0062] Figure 10 This is the seventh flowchart illustrating an intelligent optimization method for oven parameters in a film wrapping machine, as described in one embodiment.
[0063] Figure 11 This is a structural block diagram of an intelligent optimization device for oven parameters of a film wrapping machine in one embodiment. Detailed Implementation
[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0065] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0066] With the continuous development of automated packaging technology, heat shrink film packaging machines (i.e., film wrapping machines) have been widely used in the bundled packaging of bottled products in industries such as beverages, food, and daily chemicals. The film wrapping machine heats the heat shrink film placed on the outside of the bottle bundle using an oven, causing the film to shrink and tightly adhere to the surface of the bottle bundle, thus achieving bundled packaging. In actual production, multiple process parameters, such as the oven's temperature field distribution, hot air circulation volume, conveyor chain speed, and cooling air volume, jointly determine the film's shrinkage effect and the packaging yield.
[0067] In existing technologies, the parameter adjustment of the oven in a film wrapping machine is usually done manually based on experience. Specifically, the operator manually sets control parameters such as oven temperature, duct fan opening, and conveyor belt speed based on operating conditions such as film material and thickness, bottle size, and production line speed, and then repeatedly adjusts them during trial production based on shrinkage effect and yield.
[0068] However, due to the diversity of input conditions such as film material, thickness, shrinkage rate, bottle shape, and packaging size, the optimal shrinkage temperature varies significantly for different film materials. Therefore, different bottle types have different requirements for heat field distribution. In existing technologies, manual readjustment is required every time product specifications are changed, severely impacting production line efficiency.
[0069] In view of the above-mentioned technical problems, this application provides an intelligent optimization method for the oven parameters of a film wrapping machine. The following embodiments will specifically illustrate the intelligent optimization method for the oven parameters of the film wrapping machine.
[0070] The intelligent optimization method for oven parameters of a film wrapping machine provided in this application embodiment can be applied to, for example... Figure 1 The computer device shown includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores current physical parameter data, optimized control parameters, and historical performance data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an intelligent optimization method for the parameters of a membrane packaging machine oven.
[0071] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0072] In one exemplary embodiment, such as Figure 2 As shown, an intelligent optimization method for oven parameters in a film wrapping machine is provided, which is then applied to... Figure 1 Taking computer devices as an example, the explanation includes:
[0073] S201, Obtain the current physical parameters of the film wrapping machine oven to be optimized.
[0074] The current physical parameters represent all known operating conditions of the film wrapping machine oven to be optimized. The computer equipment uses the current physical parameters as input data for the optimization process to determine the optimized control parameters that match the current operating conditions.
[0075] In this embodiment, the computer device can obtain current physical parameters in the following ways: For example, it can communicate with the control system of the film packaging machine oven via an industrial Ethernet or fieldbus to read the product specification parameters and production line operating parameters corresponding to the current production batch from the control system; or the computer device can receive the current batch product information input by the operator through a human-machine interface; or the computer device can connect to the data interface of the production line management system to obtain the specification information and process parameters of the current batch of products. Specifically, the current physical parameters include film property parameters, bottle type parameters, packaging size parameters, and production line configuration parameters.
[0076] Thin film properties include at least the film material, thickness, roll diameter, longitudinal shrinkage rate, transverse shrinkage rate, and shrinkage temperature, which characterize the thermal shrinkage properties of the shrink film currently in use. Different materials, thicknesses, and shrinkage rates require different amounts of heat to achieve optimal shrinkage; therefore, these properties need to be used as input parameters in the optimization process to suit different films.
[0077] The bottle type parameters include at least the bottle type and shape, used to characterize the physical morphological features of the bottle to be packaged. For example, the type may include polyethylene terephthalate (PET) bottles, glass bottles, or aluminum cans; the shape may include round bottles, square bottles, or irregularly shaped bottles.
[0078] Packaging size parameters include at least the length, width, and height of the bottle group, which are used to characterize the overall external dimensions of a single packaged product. Packaging size directly affects the wrapping area of the film and the heat field coverage of the oven.
[0079] Production line configuration parameters include at least the number of channels and production speed of the film packaging machine oven, used to characterize the current operating status of the production line. For example, the number of channels can be single or dual; the production speed can be 30 to 90 packs / minute. Since faster production speeds result in shorter residence times of the bottle packs in the oven, the required heat compensation will also change accordingly. Therefore, production line configuration parameters also need to be used as inputs to the optimization process to adapt to different production line configurations.
[0080] It should be noted that the current physical parameters are obtained online by the computer equipment at the beginning of each optimization cycle using the above method, in order to ensure that the optimization control parameters match the current actual working conditions.
[0081] S202, the current physical parameters are input into the pre-trained target feature extraction network to obtain the shared feature vector.
[0082] In this context, the shared feature vector refers to the low-dimensional, dense feature representation extracted by the target feature extraction network after the current physical parameters are input. Furthermore, the shared feature vector can be used as input to the target policy network, enabling it to output corresponding optimized control parameters based on this shared feature vector. Simultaneously, the shared feature vector can also be used as input to other network modules (such as the dual-evaluation network during the training phase).
[0083] In this embodiment, after obtaining the current physical parameters, the computer device classifies the numerical and non-numerical parameters into categories. Then, it performs data preprocessing on both categories, concatenating the preprocessed parameters to obtain the current state vector. This current state vector is then input into a target feature extraction network. This network performs nonlinear transformations and dimensionality reduction on the current state vector to extract a shared feature vector that characterizes the essential features of the current operating condition. The target feature extraction network is a pre-trained multi-layer fully connected neural network that maps the high-dimensional current state vector to a low-dimensional shared feature vector through multi-layer nonlinear transformations. It should be noted that the shared feature vector is an intermediate representation obtained by the target feature extraction network after extracting features from the current physical parameters. It is itself a numerical vector and does not directly possess physical meaning, but the activation value of each dimension corresponds to the response intensity of the current operating condition in different feature dimensions.
[0084] S203, the shared feature vector is input into the pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven.
[0085] The target feature extraction network and target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target. Optimized control parameters refer to the multi-dimensional adjustable parameters output by the computer equipment after making decisions about the current operating conditions through the target policy network, used to control the operation of the film wrapping machine's oven. These optimized control parameters include the duct fan blade opening ratio, conveyor belt speed, target heating tube temperature, side fan disc movement distance, air vent opening ratio, and cooling fan motor speed.
[0086] In this embodiment, after obtaining the shared feature vector, the computer device inputs the shared feature vector into a pre-trained target policy network. The target policy network, based on the current operating condition characteristics represented by the shared feature vector, calculates a set of optimized control parameters matching the current operating condition through forward propagation. In this system, the value of each dimension in the optimized control parameters corresponds to the control quantity of an actuator. The duct fan blade opening ratio (0-100%) is used to control the air volume of each duct; the conveyor belt speed is used to control the speed (m / min) of the bottle group passing through the oven or the frequency value (Hz) of the frequency converter; the heating element target temperature is used to control the target heating temperature of the oven; the side fan plate moving distance and the air vent opening ratio are used together to adjust the heat field distribution inside the oven (the side fan plate moving distance is used to control the moving speed of the two side air boxes; the air vent opening ratio is used to control the opening ratio of the baffle plate (0-100%)); and the cooling fan motor speed is used to control the cooling effect.
[0087] After obtaining the optimized control parameters, the computer equipment sends these parameters to the programmable logic controller (PLC) or actuator of the film wrapping oven, causing the oven to operate according to these parameters. It should be noted that the target policy network is pre-trained by the computer equipment using a dual-evaluation network through multi-objective reinforcement learning. During training, the first objective function of the dual-evaluation network corresponds to the shrinkage effect objective, and the second objective function corresponds to the yield rate objective. Through joint optimization of these two objective functions, the trained target policy network achieves a balance between shrinkage effect and yield rate, maximizing shrinkage effect while ensuring yield rate.
[0088] In the aforementioned intelligent optimization method for the oven parameters of the film packaging machine, the current physical parameters of the oven to be optimized are obtained; these parameters are then input into a pre-trained target feature extraction network to obtain a shared feature vector; and the shared feature vector is input into a pre-trained target strategy network to obtain the optimized control parameters of the oven. The target feature extraction network and the target strategy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect objective and a second objective function corresponding to the yield rate objective. This method uses the dual evaluation network to evaluate the value of both the shrinkage effect and the yield rate objectives and jointly optimizes the strategy network, achieving an adaptive balance between shrinkage effect and yield rate. Simultaneously, by inputting the current physical parameters into the strategy network to output optimized control parameters, it achieves rapid response to various operating conditions such as different film materials, bottle types, packaging sizes, and production line speeds, eliminating the need for repeated manual adjustments. This not only reduces scrap rate and energy consumption but also further improves the production line efficiency.
[0089] In an exemplary embodiment, the phrase "inputting the current physical parameters into a pre-trained target feature extraction network to obtain a shared feature vector" in S202 above is as follows: Figure 3 As shown, it includes:
[0090] S301, obtain the optimized control parameters and historical performance data of the target policy network output in the previous round.
[0091] Historical performance data refers to the performance statistics generated by the film wrapping machine oven during its historical operation prior to the current optimization cycle. Historical performance data may include shrinkage effect indicators and yield rate statistics. The shrinkage effect indicator characterizes the tightness, uniformity, and surface wrinkling of the film shrinkage during historical production; the yield rate statistics characterize the proportion of qualified products to the total production quantity during historical production.
[0092] In this embodiment of the application, when the computer device performs optimization in the current round, it first reads the optimization control parameters output by the target policy network in the previous round from the local memory or cache. And reading historical performance data from the experience queue. If this is the first execution, meaning there are no optimized control parameters from the previous round, the computer can set the optimized control parameters from the previous round to preset initial values, such as setting each control parameter to 50% opening or preset default process parameters.
[0093] For historical performance data Historical performance data refers to the performance statistics generated by the membrane wrapping machine oven during its historical operation prior to the current optimization cycle. Specifically, historical performance data includes the most recent K sets of historical data, yield statistics, and shrinkage effect index data. Among them, the most recent K sets of historical data are the complete trajectory data of the most recent K experiments in the experience queue, including the state vector, control parameter action data, reward data, and the state vector at the next moment for each experiment, which are used to reflect the effectiveness of the recent control strategy.
[0094] The yield rate statistics include yield rate trend data, yield rate mean data, and yield rate variance data. The yield rate trend data is denoted as... ,and ,in, and Representing the most recent Group and The average yield rate of the data set. The average yield rate is denoted as... Used to characterize the average historical yield rate; the yield rate variance data is denoted as... Used to characterize the degree of fluctuation in historical yield rate.
[0095] The shrinkage effect index data is denoted as ( ),in The dimensions representing the shrinkage effect index are used to characterize the tightness, uniformity, and surface wrinkles of the film shrinkage during historical production processes. The better the shrinkage effect, the better the current control parameters.
[0096] Computer equipment will display yield rate trend data Average yield data And yield variance data data on shrinkage effect indicators Combined to form historical performance data ( It should be noted that historical performance data enables the target feature extraction network to perceive the trend and stability of historical control effects when extracting shared features, thereby assisting the target policy network in making more reasonable decisions.
[0097] S302, normalize the numerical parameters in the current physical parameters to obtain the first input parameter, and embed the non-numerical parameters in the current physical parameters to obtain the second input parameter.
[0098] In this embodiment of the application, since the current physical parameters include both non-numerical parameters and numerical parameters, the computer device needs to perform different preprocessing on the two types of parameters respectively.
[0099] For non-numerical parameters, there are three categories of discrete parameters: film material, bottle type, and bottle shape. The number of film material categories is denoted as L, the number of bottle type categories as M, and the number of bottle shape categories as N. The computer device uses embedding encoding to create an independently trainable embedding matrix for each category of discrete parameters, which is designated as the first embedding matrix. ; Second embedding matrix and the third embedding matrix In this matrix, the number of rows corresponds to the number of categories of that category, and the number of columns is the preset embedding dimension. For example, if the embedding dimension of a company is 3-dimensional, then the format of each embedding dimension is as shown in relation (1):
[0100] (1);
[0101] For example, for thin film materials Types of computer devices from the first embedded matrix Get the first place Row vectors yield the 3D dense vector representation of the category, denoted as Similarly, bottle type and bottle shape are respectively determined through the second embedding matrix. and the third embedding matrix The mapping is to a corresponding 3D dense vector. The embedding matrix is a trainable parameter that is updated synchronously through backpropagation during model training to learn the semantic similarity between categories. After embedding encoding, the three discrete parameters yield a total of 9-dimensional embedding feature vectors.
[0102] The numerical parameters include seven continuous parameters: film thickness, film roll diameter, longitudinal shrinkage rate, transverse shrinkage rate, shrinkage temperature, production speed, and packaging dimensions (bottle group length, width, and height). The computer equipment normalizes these seven numerical parameters, mapping parameters of different dimensions to the same numerical range. Subsequently, the computer equipment concatenates the nine-dimensional embedded feature vector obtained from the embedding encoding with the seven-dimensional numerical feature vector obtained from the normalization process, resulting in a 16-dimensional preprocessed feature representation. This 16-dimensional feature is then input as the feature representation of the current physical parameters into subsequent steps. It should be noted that the dimensions of the embedding matrix, the normalization method of the numerical parameters, and the feature concatenation order described above are merely illustrative examples and can be adjusted according to specific needs in practical applications. This application embodiment does not limit these aspects.
[0103] S303, the first input parameter, the second input parameter, the optimized control parameter output from the previous round, and the historical performance data are concatenated to obtain the current state vector, and the current state vector is input into the target feature extraction network to obtain the shared feature vector.
[0104] In this embodiment, after obtaining the first input parameter (i.e., the normalized numerical parameter), the second input parameter (the embedded and encoded parameter), the optimized control parameters output from the previous round, and historical performance data, the computer device concatenates these four parts of data into a vector according to a preset order to obtain the current state vector. Specifically, the first input parameter is a 7-dimensional numerical feature vector, including film thickness, film roll diameter, longitudinal shrinkage rate, transverse shrinkage rate, shrinkage temperature, production speed, and packaging size; the second input parameter is a 9-dimensional embedded feature vector, including the film material, bottle type, and bottle shape, each with 3-dimensional features after embedded encoding; the optimized control parameters output from the previous round are a 6-dimensional vector, including the duct fan blade opening ratio, conveyor belt speed, heating tube target temperature, side fan disc moving distance, air vent opening ratio, and cooling fan motor speed; the historical performance data is an m+3-dimensional vector, including yield rate trend data, yield rate mean data, yield rate variance data, and shrinkage effect index data. After concatenating the above four parts of data, the computer device obtains the current state vector s = (input parameter...). (Including the first and second input parameters), the optimized control parameters output from the previous round. Historical performance data If the current state vector has a total dimension of 16 + 6 + (m + 3) = m + 25, then the current state vector fully contains the physical characteristics of the current operating condition, the trend information of historical control effects, and the control decisions of the previous round, providing a comprehensive data foundation for subsequent feature extraction. It should be noted that the above splicing order is only an illustrative example. In practical applications, the splicing order can be adjusted according to specific needs, as long as the modules can correspond correctly. This application embodiment does not limit this.
[0105] Next, the computer device inputs the current state vector into the target feature extraction network. The target feature extraction network is a pre-trained multi-layer fully connected neural network, which performs feature extraction and dimensionality reduction on the current state vector through multi-layer nonlinear transformations, mapping the high-dimensional current state vector into a low-dimensional shared feature vector.
[0106] In an exemplary embodiment, the training process of the target feature extraction network and the target policy network described above is as follows: Figure 4 As shown, it includes:
[0107] S401, Obtain historical experimental data.
[0108] In this embodiment, historical experimental data refers to a set of historical production data samples pre-collected by computer equipment during the model training phase for training the intelligent optimization model. This data primarily includes multiple sets of physical parameter samples and control parameter action sample data. Specifically, each set of physical parameter samples corresponds to the operating conditions data in a historical experiment, which may include film property parameters, bottle type parameters, packaging size parameters, and production line configuration parameters. Each set of control parameter action samples corresponds to the control parameters actually used under those operating conditions in a historical experiment, including the duct fan blade opening ratio, conveyor belt speed, heating tube target temperature, side fan disc movement distance, air vent opening ratio, and cooling fan motor speed.
[0109] It should be noted that computer equipment can acquire historical experimental data in various ways. For example, product quality in historical production batches can be inspected and recorded manually; or visual inspection equipment can automatically collect film shrinkage effect data and yield rate data; or historical control parameters and corresponding production result data can be read from the historical operation database of the film wrapping machine oven. This application does not limit the specific method of acquiring historical experimental data.
[0110] The computer equipment uses the acquired historical experimental data as a training sample set for subsequent joint training of the basic feature extraction network, basic policy network, and dual evaluation network.
[0111] S402, determine the sample state vector based on historical experimental data, and input the sample state vector into the basic feature extraction network to obtain the basic state vector; input the basic state vector into the basic policy network to obtain the basic control parameters, and substitute the basic state vector into the first objective function and the second objective function of the dual evaluation network to obtain the first state parameter and the second state parameter.
[0112] In this embodiment, the computer device first determines the sample state vector based on historical experimental data. Specifically, for each set of historical experimental data, the computer device normalizes the numerical parameters in the physical parameter samples and performs embedding encoding on the non-numerical parameters. Then, it concatenates the normalized numerical parameters, the embedded non-numerical parameters, the corresponding historical control parameter action samples, and the historical performance statistics to obtain the sample state vector corresponding to that set of historical experimental data. The dimension of the sample state vector is consistent with the dimension of the state vector in the inference stage, both being m+25 dimensions. Next, the computer device inputs the sample state vector into the basic feature extraction network to obtain the basic state vector. The basic feature extraction network is a multi-layer fully connected neural network, and the computer device sets the dimension of the input sample state vector to D=m+25; the number of hidden neurons in the first layer is... The output dimension of the second layer is The weight parameters for the first layer are: The bias parameters for the first layer are: The weight parameters for the second layer are: And the bias parameters of the second layer are Therefore, the linear transformation and activation of the first layer can be represented by relation (2) and relation (3) respectively, as shown below:
[0113] (2);
[0114] (3);
[0115] in, Yes Perform the ReLU operation element by element.
[0116] The linear transformation and activation of the second layer can be represented by relation (4) and relation (5) respectively, as shown below:
[0117] (4);
[0118] (5);
[0119] in, The basic state vector (i.e., shared features) output by the basic feature extraction network has a dimension of . .
[0120] Subsequently, the computer device inputs the basic state vectors into the basic policy network and the dual evaluation network, respectively. For the basic policy network (such as...) Figure 5 As shown), it receives the basic state vector. Then, the mean of the Gaussian distribution of the action is output through two independent fully connected layers. ( ) and standard deviation ( The calculation method can be represented by relation (6) and relation (7) respectively, as shown below:
[0121] (6);
[0122] (7);
[0123] in, This represents the weight parameters of the first fully connected layer; This represents the weight parameters of the second fully connected layer; This represents the bias term of the first fully connected layer; This represents the bias term of the second fully connected layer.
[0124] Thus, the computer device can obtain the Gaussian distribution of the actions of the basic policy network. The computer equipment can sample the basic control parameters from this Gaussian distribution. The basic control parameters are 6-dimensional vectors, including the duct fan blade opening ratio, the mesh belt speed, the target temperature of the heating tube, the side fan disk moving distance, the air vent opening ratio, and the cooling fan motor speed.
[0125] For dual-evaluation networks (such as...) Figure 6 As shown), it receives the basic state vector. Then, the first state parameter and the second state parameter are output through two independent fully connected layers, respectively. The first fully connected layer outputs the first state parameter, which can be calculated by the relation (8), as shown below:
[0126] (8);
[0127] in, Indicates the weighting parameter; Indicates the bias term; This indicates the output value.
[0128] The second fully connected layer outputs the second state parameter, which can be calculated using the relation (9), as shown below:
[0129] (9);
[0130] The first state parameter is used to evaluate the expected return of the current strategy on the shrinkage effect target, and the second state parameter is used to evaluate the expected return of the current strategy on the yield rate target. It should be noted that the basic policy network and the dual evaluation network share the basic state vector output by the basic feature extraction network. .
[0131] S403 inputs the basic control parameters into the membrane packaging machine oven to obtain the initial reward data and the sample state vector for the next moment.
[0132] The initial reward data includes first reward data for measuring the shrinkage effect and second reward data for measuring the yield rate.
[0133] In this embodiment, the computer device sends basic control parameters to the programmable logic controller (PLC) or actuator of the shrink-wrapping oven, enabling a complete experimental batch to run according to the basic control parameters. The computer device converts the basic control parameters (duct fan blade opening ratio, conveyor belt speed, heating element target temperature, side fan disc movement distance, vent opening ratio, and cooling fan motor speed) into control instructions recognizable by the PLC, and sends these instructions to the PLC controller of the shrink-wrapping oven. The PLC controller then drives the corresponding actuator to operate according to the received control instructions. After the shrink-wrapping oven has run a complete experimental batch according to the basic control parameters, the computer device collects quality inspection data of the batch of products through a detection device, and calculates initial reward data (including first reward data and second reward data) based on the quality inspection data. For example, the detection device includes at least a visual inspection device and a dimensional measurement device for inspecting the products that have completed shrink packaging.
[0134] The first reward data is used to measure the shrinkage effect and can be calculated based on at least one of the following indicators: the tightness of the adhesion between the film and the bottle surface, the uniformity of shrinkage, the number and area of surface wrinkles, and the degree of film damage. For example, a computer device acquires a product image collected by a vision inspection device, then performs image analysis on the product image, extracts shrinkage effect-related indicators, and calculates the value of the first reward data according to a preset scoring rule. The better the shrinkage effect, the higher the value of the first reward data.
[0135] The second reward data is used to measure the yield rate and can be calculated based on the proportion of qualified products in the experimental batch to the total number of products. For example, a computer device acquires the product quality judgment results output by the testing device, then counts the number of qualified products and the total number of products. Dividing the number of qualified products by the total number of products yields the value of the second reward data, which ranges from 0% to 100%. The higher the yield rate, the higher the value of the second reward data.
[0136] While acquiring the initial reward data, the computer equipment also needs to acquire the sample state vector for the next time step. Specifically, after the membrane roll oven executes the basic control parameters and completes one experimental batch, the current physical parameters of the membrane roll oven may change (e.g., the membrane roll diameter decreases due to experimental consumption). The computer equipment then re-acquires the physical parameters at this time step, preprocesses and splices the re-acquired physical parameters to obtain the sample state vector for the next time step. This sample state vector for the next time step is used together with the current sample state vector, the basic control parameters, and the initial reward data to form a complete trajectory data, which is then stored in the experience queue.
[0137] It should be noted that if the physical parameters of the membrane wrapping machine oven do not change before and after execution in the current round, the sample state vector at the next moment can be the same as the current sample state vector. This application embodiment does not limit this.
[0138] S404 stores the sample state vector, basic control parameters, initial reward data, and the sample state vector at the next moment as a trajectory data into the experience queue.
[0139] In this embodiment, after acquiring the initial reward data and the sample state vector for the next moment, the computer device combines the sample state vector generated in the current round, the basic control parameters, the initial reward data, and the sample state vector for the next moment into a complete trajectory data, and stores this trajectory data in an experience queue. Each trajectory data record records the complete process information from starting from a certain state, taking a certain action, obtaining the corresponding reward, and moving to the next state. The experience queue is used to store the most recent trajectory data, and its capacity is a preset threshold K, where K is a positive integer (e.g., K=20). For example, when the number of trajectories already stored in the experience queue has not reached the preset threshold, the computer device can continuously store newly generated trajectory data into the experience queue until it is full (K trajectory data); when the number of trajectories already stored in the experience queue reaches the preset threshold, the computer device can store newly generated trajectory data at the end of the experience queue and delete the trajectory data with the longest storage time to maintain the number of trajectories in the experience queue at always being K.
[0140] S405, when the number of trajectories in the experience queue reaches a preset threshold, for each trajectory data in the experience queue, the first advantage function value and the second advantage function value are calculated respectively based on each initial reward data, each first state parameter and each second state parameter.
[0141] In this embodiment of the application, for the t-th trajectory data in the experience queue, the computer device obtains the initial reward data for that trajectory data. (in This represents the primary reward data for measuring the effectiveness of contraction. (This represents the second reward data used to measure yield rate), and the sample state vector of this trajectory data. The sample state vector at the next time step The first state parameter corresponding to this trajectory data Second state parameters The dual-evaluation network is the sample state vector for the next time step. First state parameter output And the dual-evaluation network is the sample state vector for the next time step. Output second state parameter It is worth noting that in the early stages of training, because the experience queue is not yet full, the computer does not calculate the advantage function or update the network; this phase is the cold start period. During the cold start period, the computer only performs data collection and storage, and the reward data can be set to 0 or a preset initial value.
[0142] Subsequently, for the t-th trajectory data, the computer device can perform the first time-series difference error. Second timing difference error The first time-series difference error is calculated. It can be represented by relation (10), which is shown below:
[0143] (10);
[0144] in, This represents a preset discount factor used to control the weight of rewards, and .
[0145] The second timing difference error It can be represented by relation (11), which is shown below:
[0146] (11);
[0147] After calculating the temporal difference error for each trajectory, the computer calculates the dominance function value by accumulating the multi-step temporal difference error. The first dominance function value... The calculation method can be represented by relation (12), which is shown below:
[0148] (12);
[0149] Second advantage function value The calculation method can be represented by relation (13), which is shown below:
[0150] (13);
[0151] in, This represents the preset smoothing factor, and ; This represents the preset discount factor, whose value is consistent with the discount factor value in the aforementioned time series difference error.
[0152] Through the above calculations, the computer equipment obtains a first dominance function value and a second dominance function value for each trajectory data point in the experience queue. The first dominance function value measures the superiority or inferiority of the action taken in the current state relative to the average level in terms of the contraction effect target, while the second dominance function value measures the superiority or inferiority of the action taken in the current state relative to the average level in terms of the yield target. A positive dominance function value indicates that the action is better than the average level; a negative value indicates that the action is worse than the average level. It should be noted that the specific values of the discount factor and smoothing factor can be set according to actual needs.
[0153] S406, update the basic policy network, dual evaluation network, and basic feature extraction network according to the values of each first advantage function and each second advantage function, until the reward data, shrinkage effect, and yield rate all meet the expected goals, and obtain the target feature extraction network and target policy network.
[0154] In this embodiment, after obtaining the values of each first advantage function and each second advantage function, the computer device can update the basic policy network, the dual evaluation network, and the basic feature extraction network based on these advantage function values. In one update iteration, the three networks adjust their parameters according to their respective methods and directions. For example, the basic policy network is updated along the policy gradient direction to improve policy performance, the dual evaluation network is updated along the evaluation gradient direction to improve the accuracy of value estimation, and the basic feature extraction network is updated along the weighted fusion direction of the three gradients to improve the representational ability of shared features. After each update, the computer device determines whether the current model meets the preset convergence conditions. Specifically, the computer device obtains the reward data, shrinkage effect index, and yield rate index of the current policy on the validation dataset and determines whether the above indicators all meet the expected goals. For example, the expected goals include: the shrinkage effect reward reaches or exceeds a preset threshold (e.g., shrinkage effect score ≥ 90 points), and the yield rate reaches or exceeds a preset threshold (e.g., yield rate ≥ 98%), and the fluctuation range of the above indicators does not exceed a preset range within multiple consecutive training iterations (e.g., 10 consecutive cycles). When the above conditions are met simultaneously, the computer determines that the model has converged and stops training. When training stops, the computer designates the current basic feature extraction network as the target feature extraction network. During the inference phase, the computer sequentially inputs the real-time collected physical parameters into the target feature extraction network and the target policy network to obtain the optimized control parameters for the membrane packaging machine oven under the current operating conditions. It is worth noting that the inference phase no longer involves training-related steps such as dual evaluation networks, experience queues, advantage function calculation, and network parameter updates.
[0155] It should be noted that although the target dual-evaluation network obtained after training does not participate in the inference process, it can be retained for subsequent model fine-tuning or performance evaluation. When incremental training of the model is required or to adapt to new working conditions, training can continue based on the target feature extraction network, the target policy network, and the target dual-evaluation network.
[0156] In an exemplary embodiment, the process of updating the basic policy network described above is as follows: Figure 7 As shown, it includes:
[0157] S501, the first policy gradient and the second policy gradient are calculated based on the values of each first advantage function and each second advantage function.
[0158] The historical experimental data includes multiple sets of physical parameter samples and control parameter action sample data.
[0159] In this embodiment of the application, after obtaining the first dominance function value and the second dominance function value corresponding to each trajectory in the experience queue, the computer device first calculates the importance sampling ratio. Calculate the importance sampling ratio. This indicates that the current policy and the old policy are in the same state. Take the same action The probability ratio can be expressed by the relation (14), which is shown below:
[0160] (14);
[0161] in, Indicates the current state of the policy network. Take action below The probability density; This indicates that the old policy network before the update is in the same state. Take the same action The probability density. The importance sampling ratio is used to measure the degree of difference between the new and old strategies, when... The greater the deviation from 1, the larger the policy update magnitude.
[0162] Subsequently, the computer equipment calculates the first policy objective function and the second policy objective function, respectively. The first policy objective function... It can be represented by relation (15), which is shown below:
[0163] (15);
[0164] Second strategy objective function It can be represented by relation (16), which is shown below:
[0165] (16);
[0166] in, .
[0167] The computer devices respectively target the first strategy objective function Second strategy objective function By performing gradient calculation, the gradient of the first policy can be obtained. Second policy gradient Among them, the gradient of the first policy It can be represented by relation (17), which is shown below:
[0168] (17);
[0169] Second policy gradient It can be represented by relation (18), which is shown below:
[0170] (18);
[0171] in, The gradient represents the logarithmic probability of the policy, indicating the direction of the effect of changes in policy parameters on the action probability. First policy gradient. The second policy gradient is used to indicate which direction the basic policy network updates in order to improve the contraction objective. It is used to indicate in which direction the basic policy network updates to improve the yield target.
[0172] S502, when there is a directional conflict between the first policy gradient and the second policy gradient, the first policy gradient and the second policy gradient are projected based on the projection conflict gradient method.
[0173] In this embodiment of the application, the computer device obtains the first policy gradient. Second policy gradient Next, the system first checks for directional conflict between the two gradients. This involves the computer calculating the dot product of the first and second policy gradients. If the dot product is less than 0, then... If a direction conflict is detected, the computer device determines that the first policy gradient and the second policy gradient have a direction conflict. In the case of detected gradient direction conflict, the computer device projects the first policy gradient and the second policy gradient respectively based on the projection conflict gradient method. Specifically, the projection processing of the first policy gradient by the computer device can be represented by relation (19), which is shown below:
[0174] (19);
[0175] The projection processing of the second policy gradient by the computer device can be represented by relation (20), which is shown below:
[0176] (20);
[0177] in, express Norm.
[0178] It should be noted that if the gradients of the first policy and the second policy do not conflict in direction (i.e.) If the computer does not project the two gradients, it will directly retain the original gradients for subsequent updates.
[0179] S503 updates the base policy network based on the first policy gradient and the second policy gradient after projection processing.
[0180] In this embodiment, the computer device averages the first and second policy gradients after projection processing to obtain the final policy gradient used to update the basic policy network. The policy gradient It can be represented by relation (21), which is shown below:
[0181] (twenty one);
[0182] Subsequently, the computer equipment follows the policy gradient. Update the parameters of the basic policy network, i.e. ;in This represents the parameters of the updated base policy network. This represents the parameters of the base policy network before the update. This indicates the preset policy network step size.
[0183] Through the above updates, the basic policy network adjusts its parameters in a direction that simultaneously improves the contraction effect and yield rate, enabling the updated policy network to achieve a higher overall reward for the control parameters output in the current state. It should be noted that the computer device's update of the basic policy network is based on one iteration of updates to K trajectory data points in the experience queue.
[0184] In an exemplary embodiment, the above-described process of updating the dual-evaluation network is as follows: Figure 8 As shown, it includes:
[0185] S601, based on each first dominant function value and each second dominant function value, obtain the first value loss value and the second value loss value respectively.
[0186] In this embodiment, after calculating the values of each first dominance function and each second dominance function, the computer device can calculate the first value loss and the second value loss based on these dominance function values, and use them to update the dual evaluation network. The dual evaluation network includes a first evaluation network and a second evaluation network, where the first evaluation network corresponds to the contraction effect target and the second evaluation network corresponds to the yield rate target. Specifically, for K trajectory data in the experience queue, the computer device takes the first state parameter corresponding to each trajectory. Second state parameters The first value loss of the first evaluation network can be represented by relation (22), which is shown below:
[0187] (twenty two);
[0188] The second value loss of the second evaluation network can be represented by relation (23), which is shown below:
[0189] (twenty three);
[0190] It should be noted that the first evaluation network and the second evaluation network are two independent networks, each calculating its own value loss, and they do not affect each other.
[0191] S602, determine the first evaluation gradient and the second evaluation gradient based on the first value loss value and the second value loss value respectively, and update the dual evaluation network based on the first evaluation gradient and the second evaluation gradient.
[0192] In this embodiment of the application, after obtaining the first value loss value and the second value loss value, the computer device can obtain the first evaluation gradient and the second evaluation gradient according to the relation (24) and the relation (25). The relation (24) and the relation (25) are as follows:
[0193] (twenty four);
[0194] (25);
[0195] in, The gradient of the network parameters in the first evaluation network output represents the direction of the influence of changes in network parameters on the value estimation output. This represents the gradient of the network parameters output by the second evaluation network.
[0196] Subsequently, after the computer device obtains the first evaluation gradient and the second evaluation gradient, the parameters of the first evaluation network and the second evaluation network can be updated according to the first evaluation gradient and the second evaluation gradient, respectively. The parameter update of the first evaluation network can be represented by relation (26), which is shown below:
[0197] (26);
[0198] The parameter update of the second evaluation network can be represented by relation (27), which is shown below:
[0199] (27);
[0200] in, and These are the parameters of the updated first and second evaluation networks, respectively. This represents the preset learning rate for the evaluation network, used to control the step size of each parameter update. (Evaluation network learning rate) With policy network learning rate The values can be the same or different; this application's embodiments do not limit this. It should be noted that the first and second evaluation networks are updated independently and do not affect each other. The updated first evaluation network can be used for value estimation of the shrinkage effect target in the next round of training, and the updated second evaluation network can be used for value estimation of the yield rate target in the next round of training. After each update of the dual evaluation networks, the updated dual evaluation network parameters can be saved to local storage or cache for use in subsequent training iterations.
[0201] In an exemplary embodiment, the process of updating the basic feature extraction network described above is as follows: Figure 9 As shown, it includes:
[0202] S701 determines the policy gradient when updating the basic policy network based on the first and second policy gradients after projection processing.
[0203] In this embodiment, the computer device averages the first and second policy gradients after projection processing to obtain the final policy gradient, i.e., relation (21). It should be noted that the determined final policy gradient... With the final policy gradient used to update the base policy network This is the same gradient. That is, the computer device first utilizes this gradient. Update the base policy network and obtain the gradient again. This is used for updating the subsequent basic feature extraction network.
[0204] S702 determines the feature extraction gradient based on the policy gradient, the first evaluation gradient, and the second evaluation gradient, and updates the basic feature extraction network according to the feature extraction gradient.
[0205] In this embodiment of the application, the computer device obtains the policy gradient. Then, by combining the first and second evaluation gradients, the feature extraction gradient of the basic feature extraction network can be determined. Specifically, the policy gradient... After inverting, the gradient is weighted and summed with the first and second evaluation gradients to obtain the feature extraction gradient. The feature extraction gradient It can be represented by relation (28), which is shown below:
[0206] (28);
[0207] in, and These are preset weight coefficients, and satisfy... ; This indicates the degree of contribution of the first evaluation gradient to the feature extraction gradient update; This indicates the degree to which the second evaluation gradient contributes to the feature extraction gradient update. For example, if... =0.5, If the coefficient of variation is 0.5, then the two evaluation gradients each contribute half to the feature extraction network update. That is, the computer equipment can adjust the gradient based on the relative importance placed on shrinkage effect and yield rate in actual production. and Adjustments will be made.
[0208] Subsequently, the computer device updates the parameters of the basic feature extraction network according to the feature extraction gradient, which can be represented by relation (29), as shown below:
[0209] (29);
[0210] in, This represents the parameters of the basic feature extraction network before the update; This represents the parameters of the updated basic feature extraction network.
[0211] It should be noted that the computer device updates the basic feature extraction network based on one iteration of updates to K trajectory data in the experience queue. In the actual training process, Figure 9 The steps shown will be repeated for multiple training iterations (i.e., repeated) Figure 4(Follow the steps shown) until the reward data, shrinkage effect, and yield all meet the expected targets. After training, the base feature extraction network is determined as the target feature extraction network.
[0212] In summary, based on all the above embodiments, a method for intelligent optimization of oven parameters in a film wrapping machine is also provided, such as... Figure 10 As shown, the method includes:
[0213] S801, Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0214] S802, obtain the optimized control parameters and historical performance data of the target policy network output in the previous round;
[0215] S803, normalize the numerical parameters in the current physical parameters to obtain the first input parameter, and perform embedding encoding on the non-numerical parameters in the current physical parameters to obtain the second input parameter;
[0216] S804: The first input parameter, the second input parameter, the optimized control parameter output from the previous round, and the historical performance data are concatenated to obtain the current state vector. The current state vector is then input into the target feature extraction network to obtain the shared feature vector. S806-S818 are executed to train the target feature extraction network and the target policy network.
[0217] S805, the shared feature vector is input into the pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0218] S806, acquire historical experimental data;
[0219] S807: Determine the sample state vector based on historical experimental data, and input the sample state vector into the basic feature extraction network to obtain the basic state vector; input the basic state vector into the basic policy network to obtain the basic control parameters; and substitute the basic state vector into the first objective function and the second objective function of the dual evaluation network to obtain the first state parameter and the second state parameter.
[0220] S808 inputs basic control parameters into the membrane packaging machine oven to obtain initial reward data and the sample state vector for the next moment;
[0221] S809 stores the sample state vector, basic control parameters, initial reward data, and the sample state vector at the next moment as a trajectory data into the experience queue.
[0222] S810, when the number of trajectories in the experience queue reaches a preset threshold, calculate the first advantage function value and the second advantage function value for each trajectory data in the experience queue based on each initial reward data, each first state parameter and each second state parameter respectively;
[0223] S811, update the basic policy network, dual evaluation network, and basic feature extraction network based on the values of each first advantage function and each second advantage function, until the reward data, shrinkage effect, and yield rate all meet the expected targets, thus obtaining the target feature extraction network and target policy network; execute S812-S814 to update the basic policy network; execute S815-S816 to update the dual evaluation network; execute S817-S818 to update the basic feature extraction network;
[0224] S812, the first policy gradient and the second policy gradient are calculated based on each first advantage function value and each second advantage function value, respectively;
[0225] S813, when there is a directional conflict between the first policy gradient and the second policy gradient, the first policy gradient and the second policy gradient are projected based on the projection conflict gradient method.
[0226] S814, update the base policy network based on the first policy gradient and the second policy gradient after projection processing;
[0227] S815, based on each first dominant function value and each second dominant function value, the first value loss value and the second value loss value are obtained respectively;
[0228] S816, determine the first evaluation gradient and the second evaluation gradient based on the first value loss value and the second value loss value respectively, and update the dual evaluation network based on the first evaluation gradient and the second evaluation gradient;
[0229] S817, based on the first and second policy gradients after projection processing, determine the policy gradient when updating the basic policy network.
[0230] S818 determines the feature extraction gradient based on the policy gradient, the first evaluation gradient, and the second evaluation gradient, and updates the basic feature extraction network according to the feature extraction gradient.
[0231] The methods described in each of the above steps have been described in the foregoing embodiments. For details, please refer to the foregoing descriptions. They will not be repeated here.
[0232] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0233] Based on the same inventive concept, this application also provides an intelligent optimization device for the oven parameters of a film wrapping machine, which is used to implement the intelligent optimization method for oven parameters of a film wrapping machine as described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent optimization device for oven parameters of a film wrapping machine provided below can be found in the limitations of the intelligent optimization method for oven parameters of a film wrapping machine described above, and will not be repeated here.
[0234] In one exemplary embodiment, such as Figure 11 As shown, an intelligent optimization device for oven parameters of a film wrapping machine is provided, comprising:
[0235] Module 11 is used to acquire the current physical parameters of the film wrapping machine oven to be optimized;
[0236] Extraction module 12 is used to input the current physical parameters into a pre-trained target feature extraction network to obtain a shared feature vector;
[0237] Strategy module 13 is used to input the shared feature vector into the pre-trained target strategy network to obtain the optimized control parameters of the membrane packaging oven; wherein, the target feature extraction network and the target strategy network are trained using a dual evaluation network; the dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield target.
[0238] Each module in the intelligent optimization device for the oven parameters of the film wrapping machine described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0239] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0240] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0241] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0242] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0243] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0244] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0245] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0246] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0247] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0248] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0249] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0250] Obtain the current physical parameters of the film wrapping machine oven to be optimized;
[0251] The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector.
[0252] The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven;
[0253] The target feature extraction network and the target policy network are trained using a dual evaluation network. The dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
[0254] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0255] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0256] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for intelligent optimization of oven parameters in a film wrapping machine, characterized in that, The method includes: Obtain the current physical parameters of the film wrapping machine oven to be optimized; The current physical parameters are input into a pre-trained target feature extraction network to obtain a shared feature vector; The shared feature vector is input into a pre-trained target policy network to obtain the optimized control parameters of the membrane packaging machine oven; The target feature extraction network and the target policy network are trained using a dual evaluation network; the dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield rate target.
2. The method according to claim 1, characterized in that, The step of inputting the current physical parameters into a pre-trained target feature extraction network to obtain a shared feature vector includes: Obtain the optimized control parameters and historical performance data output from the previous round of the target policy network; The numerical parameters in the current physical parameters are normalized to obtain the first input parameter, and the non-numerical parameters in the current physical parameters are embedded and encoded to obtain the second input parameter; The first input parameter, the second input parameter, the optimization control parameter output from the previous round, and the historical performance data are concatenated to obtain the current state vector, and the current state vector is input into the target feature extraction network to obtain the shared feature vector.
3. The method according to claim 2, characterized in that, The training process of the target feature extraction network and the target policy network includes: Acquire historical experimental data; the historical experimental data includes multiple sets of physical parameter samples and control parameter action sample data; Based on the historical experimental data, a sample state vector is determined, and the sample state vector is input into the basic feature extraction network to obtain a basic state vector; the basic state vector is input into the basic policy network to obtain basic control parameters, and the basic state vector is substituted into the first objective function and the second objective function of the dual evaluation network to obtain the first state parameter and the second state parameter. The basic control parameters are input into the film wrapping machine oven to obtain initial reward data and the sample state vector at the next moment; the initial reward data includes first reward data for measuring the shrinkage effect and second reward data for measuring the yield rate; The sample state vector, the basic control parameters, the initial reward data, and the sample state vector at the next moment are stored as a trajectory data in the experience queue. When the number of trajectories in the experience queue reaches a preset threshold, for each trajectory data in the experience queue, a first advantage function value and a second advantage function value are calculated based on each of the initial reward data, each of the first state parameters, and each of the second state parameters, respectively. The basic policy network, the dual evaluation network, and the basic feature extraction network are updated based on the first advantage function value and the second advantage function value until the reward data, shrinkage effect, and yield rate all meet the expected targets, thus obtaining the target feature extraction network and the target policy network.
4. The method according to claim 3, characterized in that, The process of updating the basic policy network includes: The first policy gradient and the second policy gradient are calculated based on each of the first advantage function values and each of the second advantage function values, respectively. In the case where there is a directional conflict between the first policy gradient and the second policy gradient, the first policy gradient and the second policy gradient are projected respectively based on the projection conflict gradient method. The base policy network is updated based on the first policy gradient and the second policy gradient after projection processing.
5. The method according to claim 4, characterized in that, The process of updating the dual-evaluation network includes: The first value loss value and the second value loss value are obtained based on each of the first advantage function values and each of the second advantage function values, respectively. A first evaluation gradient and a second evaluation gradient are determined based on the first value loss value and the second value loss value, respectively, and the dual evaluation network is updated based on the first evaluation gradient and the second evaluation gradient.
6. The method according to claim 5, characterized in that, The process of updating the basic feature extraction network includes: Based on the first and second policy gradients after the projection processing, the policy gradient is determined when the basic policy network is updated. Based on the policy gradient, the first evaluation gradient, and the second evaluation gradient, the feature extraction gradient is determined, and the basic feature extraction network is updated according to the feature extraction gradient.
7. An intelligent optimization device for oven parameters of a film wrapping machine, characterized in that, The device includes: The acquisition module is used to acquire the current physical parameters of the film wrapping machine oven to be optimized; The extraction module is used to input the current physical parameters into a pre-trained target feature extraction network to obtain a shared feature vector; The strategy module is used to input the shared feature vector into a pre-trained target strategy network to obtain the optimized control parameters of the membrane packaging oven; wherein, the target feature extraction network and the target strategy network are trained using a dual evaluation network; the dual evaluation network includes a first objective function corresponding to the shrinkage effect target and a second objective function corresponding to the yield target.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
