Intelligent control yeast adding method and system

Abstract

The invention relates to an intelligent control yeast adding method and system. The method comprises the following steps: front-end detection: detecting state information related to a yeast adding process to obtain data information for analysis; middle-end decision making: calculating the yeast adding amount according to the acquired data information to generate a yeast adding strategy; and back-end output: corresponding yeast adding operation is executed based on the generated yeast adding strategy, in the middle-end decision making process, the potential influence of different yeast adding amounts obtained in the front-end detection process on the liquor yield and the liquor quality in the current production state is evaluated through an established and trained DQN deep reinforcement learning algorithm, and the quality of the liquor is improved. And determining an expected return of adopting a specific yeast adding amount action, so as to determine an optimal yeast adding amount and generate a yeast adding strategy for execution of a rear-end output process according to the optimal yeast adding amount. The system comprises a detection module, a control module and an execution module, and the control module can execute the method and output a corresponding control signal to the execution module to complete yeast adding operation.

Description

Technical Field

[0001] The invention relates to the technical field of automated liquor brewing, and in particular to an intelligent control method and system for adding koji. Background Art

[0002] Adding koji is one of the key links in the liquor brewing process. As a unique saccharifying and fermenting agent, koji contains rich bacterial, material and enzyme systems. It plays the role of saccharification, fermentation and aroma production in the liquor production process, giving the liquor a unique flavor. In the traditional liquor brewing process, a certain amount of koji powder is added after the fermented mash is spread out to cool down. The amount of koji powder is dynamically adjusted according to factors such as the current ambient temperature. At the same time, the amount of koji powder added must be accurately controlled to ensure the fermentation quality of the fermented mash.

[0003] CN114480048A discloses a system for spreading and drying wine lees and adding koji, which comprises a feeding device, a spreading device, a turning device, a koji-adding device and a mixing and discharging device; the spreading device comprises a chain plate conveyor belt, an air chamber and a fan; the air chamber is arranged below the chain plate conveyor belt, the air chamber is provided with an air inlet and an air outlet, the fan is connected to the air inlet of the air chamber, and the chain plate conveyor belt is provided with ventilation holes; the feeding device is arranged at the feeding end of the chain plate conveyor belt, and is used for adding wine lees to the chain plate conveyor belt; the turning device is arranged above the chain plate conveyor belt, and is used for turning over the wine lees on the chain plate conveyor belt; the koji-adding device is arranged above the chain plate conveyor belt, and is used for adding koji-adding device to the wine lees on the chain plate conveyor belt; the mixing and discharging device is arranged at the discharging end of the chain plate conveyor belt, and is used for stirring the wine lees and koji-adding device, and sending the stirred wine lees and koji-adding device to the target position.

[0004] CN106434233A discloses an intelligent control system for liquor production, including a control device, a robot retort loading system, a PLC, various pumps, valves, motors, various sensors, and various instruments. In addition, a method for intelligent control of liquor production using the intelligent control system for liquor production is disclosed, including steps such as weighing ingredients, automatically loading retorts, distilling liquor, cooling, and adding koji.

[0005] The existing technology of adding koji mainly uses volumetric measurement to control the amount of koji powder added. However, this method has several key problems. First, due to the difference in the flow speed of the mash and the koji powder, their addition process cannot be synchronized. Specifically, when the mash is finished drying and the koji powder is added, there may still be residual koji powder in the koji bucket; conversely, when the mash has not yet finished drying and the koji is added, the koji powder in the koji bucket may have been used up. This asynchronous addition method makes it impossible to achieve effective synchronous compound addition of materials, which may affect the final fermentation effect and wine quality. In addition, the operation of adding koji is usually carried out in an open environment, which will cause the koji powder to be dispersed into the air during the addition process, causing pollution to the production environment and waste of resources. These problems need to be solved urgently to improve the efficiency of liquor brewing and product quality.

[0006] In addition, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making the present invention, but due to space limitations, not all details and contents are listed in detail. However, this does not mean that the present invention does not have the characteristics of these prior arts. On the contrary, the present invention already has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art to the background technology. Summary of the invention

[0007] In view of the deficiencies of the prior art, the present invention provides an intelligent control method and system for adding music to solve at least part of the above technical problems.

[0008] The present invention discloses an intelligent control method for adding music, which comprises:

[0009] Front-end detection: detect the status information related to the song adding process to obtain data information for analysis;

[0010] Mid-range decision-making: Calculate the amount of music to be added based on the acquired data information to generate a music addition strategy;

[0011] Backend output: Execute corresponding song adding operations based on the generated song adding strategy.

[0012] Preferably, in the mid-end decision-making process, a DQN deep reinforcement learning algorithm is established and trained to evaluate the potential impact of different amounts of added koji on the wine yield and quality under the current production status obtained in the front-end detection process, so as to determine the expected return of taking a specific amount of added koji action, thereby determining the optimal amount of added koji and generating a koji addition strategy for execution by the back-end output process.

[0013] The intelligent control method for adding koji of the present invention completes the process of adding koji through front-end detection, mid-end decision-making and back-end output. First, the front-end detection stage monitors the status information related to adding koji in real time, so that the system can obtain accurate data and significantly improve the sensitivity to raw materials and production conditions. Then, in the mid-end decision-making stage, the deep reinforcement learning algorithm (DQN) is used to analyze these data to evaluate the potential impact of different amounts of koji on wine production rate and wine quality, thereby forming a dynamic strategy for adding koji and reducing the error of empirical decision-making. Finally, in the back-end output stage, accurate koji powder addition is performed according to the generated optimal koji strategy to ensure that the optimal amount of koji under specific production conditions is implemented. Through the synergy of the above three levels, the present invention realizes a closed-loop feedback mechanism from data to decision-making and then to execution, so that each step in the process of adding koji can be adjusted based on the optimal strategy, and finally significantly improves production efficiency and product quality. The present invention not only has a breakthrough significance in the traditional brewing industry, but also provides a systematic solution that can be used as a reference in the field of intelligent manufacturing.

[0014] According to a preferred embodiment, in the mid-end decision-making process, a Q network is defined and trained to approximate a Q function to quantify the expected return of taking a specific additive action under a specific production state, and then a Q' network is introduced to stabilize the learning process, wherein the state information of the current production state can be combined into a state vector for subsequent input into the Q network for learning.

[0015] The present invention can quantify the expected returns of taking different actions of adding koji under specific production conditions by defining a Q network and training it into an approximate Q function. This quantification not only provides a clear assessment of the expected effect, but also provides a reliable basis for subsequent decision-making, avoiding the blindness and uncertainty of traditional rules of thumb. Furthermore, the introduction of the Q' network strategy makes the learning process more stable and reduces fluctuations caused by overfitting. This mechanism ensures the adaptability of the model under different production conditions, allowing the system to respond quickly to environmental changes. In addition, by merging the state information of the current production state into a state vector, the Q network can efficiently process multi-dimensional data, thereby improving learning efficiency and accuracy. In summary, this progressive technical feature not only improves the intelligence level of the koji adding process, but also provides a new perspective for precise control and optimization of production, and significantly enhances the ability to control quality and output during the winemaking process.

[0016] According to a preferred embodiment, the status information of the current production status includes status information related to the production environment and / or status information related to the mash, wherein the status information related to the production environment includes the production environment temperature, the production environment humidity and / or the production environment season; the status information related to the mash includes the mash weight, the mash temperature and / or the mash humidity, and the acquired status information can be input into the Q network after being standardized or normalized.

[0017] The status information of the current production status includes multiple indicators related to the production environment and mash, such as the temperature, humidity and season of the production environment, as well as the weight, temperature and humidity of the mash. This comprehensive data collection ensures that the intelligent agent can fully understand the changes in production conditions, thereby providing rich background information for subsequent decision-making. Furthermore, the standardized or normalized state information is input into the Q network, so that data of different dimensions and ranges can be processed uniformly, which improves the sensitivity and adaptability of the model to various input signals. This data preprocessing method not only reduces the noise interference of the input data, but also promotes the convergence speed in the learning process and enhances the stability of the model. On this basis, the Q network can more accurately evaluate the expected returns of different koji addition actions under specific environmental conditions, thereby achieving more refined control and optimization. Ultimately, this series of correlation features combined with processing methods significantly improves the intelligence level of the brewing process, opens up a new path for optimizing product quality and production efficiency, and surpasses the effects that can be achieved by traditional methods.

[0018] According to a preferred embodiment, an ε-greedy strategy is used to select a bending action according to the current state, wherein the bending action is randomly selected with a probability based on the exploration rate ε or the optimal bending action is selected with a probability of 1-ε; and the experience corresponding to the selected bending action is stored in the experience playback buffer, wherein the experience includes the state information of the current production environment, the selected bending action, the immediate reward and the new state.

[0019] The present invention adopts a mechanism based on the exploration rate ε, so that the intelligent agent can balance exploration and utilization when selecting the bending action. When the exploration rate ε is set to an appropriate value, the system can not only select the currently known optimal action with a probability of 1-ε, thereby accelerating the maximization of benefits in the learning process, but also randomly select the bending action with a probability of ε to explore new possibilities. This exploration mechanism enables the system to avoid falling into the local optimal solution and improves the comprehensive exploration ability of the state space. In addition, the selected action is stored in the experience replay buffer together with the corresponding current state information, immediate reward and new state. This design enables the system to reuse these valuable experiences in subsequent learning, thereby deepening the understanding of the relationship between specific production states and bending reactions. In this way, the system realizes the ability of dynamic learning, continuously optimizes decision-making strategies, and quickly adapts to environmental changes. Finally, this series of associated feature combinations not only improves the intelligence level of the bending process, but also provides a solid foundation for optimizing production efficiency and improving product quality, surpassing the limitations of traditional decision-making methods and opening up a more efficient and flexible production mode.

[0020] According to a preferred embodiment, a batch of experiences are randomly extracted from the experience playback buffer periodically for learning and updating, and the target Q value is calculated according to the Bellman equation after sampling the experiences, wherein the target Q value is used to evaluate the expected return that can be obtained by selecting a certain amount-added action under a specific state, and to provide a benchmark for updating the Q network.

[0021] The random sampling mechanism enables the system to learn from diverse experiences, avoiding the overfitting problem caused by the time correlation of data. This mechanism ensures data independence during training and enhances the adaptability of the model to environmental changes. By using the Bellman equation to calculate the target Q value, the system can accurately evaluate the expected return of selecting a certain curvature action in a specific state, which not only provides a quantitative benchmark for Q network updates, but also promotes a deep understanding of future results. The introduction of the target Q value makes the learning process more directional, clarifies the optimization goals of the model in different states, and thus provides a scientific basis for the improvement of the strategy. This method of extracting target returns from experience makes the learning process more efficient and targeted, thereby accelerating the convergence speed of the Q network and improving the decision quality. Finally, through this series of progressive feature integration, not only the intelligent level of curvature action selection is significantly improved, but also a solid foundation is laid for the dynamic optimization of the production process, so that the system can show unprecedented flexibility and accuracy in the face of complex and changing production environments.

[0022] According to a preferred embodiment, based on the current state and the added curvature action, the current Q value is calculated by the Q network, so as to calculate the loss by minimizing the loss function according to the difference between the current Q value and the target Q value, thereby updating the parameters of the Q network by the gradient descent method or its variants.

[0023] This process uses the Q network to map the state and the added action to the corresponding Q value, which can reflect the decision value of the system in a specific state in real time. This dynamic evaluation mechanism provides strong support for policy optimization. By calculating the difference between the current Q value and the target Q value, the system can effectively identify the shortcomings of the current strategy, and then use this difference as the basis of the loss function to make precise feedback adjustments. Such policy adjustments not only rely on past experience, but also can dynamically adapt to current environmental changes, reflecting the adaptive characteristics of the learning process. The use of gradient descent or its variants to update the parameters of the Q network makes this adjustment process more efficient and stable, ensuring that the model will not be disturbed by noise or local extreme values ​​during the convergence process. Through this series of closely related feature construction, the system not only improves the accuracy and intelligence of decision-making, but also significantly accelerates the convergence speed of the learning process, thereby achieving more efficient resource allocation and optimization capabilities in complex production environments, exceeding the expected results of traditional optimization methods, and showing a new potential for intelligent decision-making.

[0024] According to a preferred embodiment, an initial exploration rate and a final exploration rate, as well as a decay factor, are set, and the exploration rate is adjusted according to the decay factor after each training cycle, thereby gradually reducing the exploration until the training is completed, wherein the trained model algorithm is evaluated using the training set to verify its reliability and stability under different environmental conditions.

[0025] The setting of the initial exploration rate enables the model to conduct extensive exploration in the early stage of training and obtain rich empirical data. This process provides a diverse background and rich strategic choices for model learning. As the training proceeds, the exploration rate is gradually reduced through the attenuation factor, so that the model can focus on optimizing decisions using existing knowledge in the later stage. This strategy of gradually reducing exploration ensures that the model can be more stably fine-tuned in the final stage without causing performance fluctuations due to excessive exploration. At the same time, the trained model is evaluated using the training set to further verify the reliability and stability of the model under different environmental conditions, ensuring that it can maintain consistent performance in real applications. This evaluation process not only provides quantitative verification of the generalization ability of the model, but also provides a basis for subsequent model improvement and parameter adjustment, thereby forming a closed-loop optimization mechanism. Finally, through the combination of this dynamic adjustment of the exploration rate and model evaluation, the present invention can significantly improve the adaptability and decision-making accuracy of the model in a complex environment, surpass the limitations of traditional methods, and show a higher level of intelligence and long-term effectiveness.

[0026] According to a preferred embodiment, during the back-end output process, the corresponding koji adding operation is performed according to the generated koji adding strategy, and the change in the koji powder weight in the koji adding container is monitored by weighing to determine the koji adding end point, wherein when the koji adding end point is reached, the koji adding container is closed to stop the koji adding operation.

[0027] The present invention implements the corresponding addition operation through the generated addition strategy, ensures the accuracy and consistency of the operation, and makes each addition process strictly follow the preset parameters, thereby optimizing the production efficiency. The introduction of the weighing method provides an effective means for real-time monitoring of the change in the weight of the koji powder in the addition container. This dynamic feedback mechanism enables the system to obtain key data in the addition process in a timely manner, and then judge whether the addition reaches the predetermined end point. This judgment method based on weight change not only improves the automation of the addition operation, but also reduces the need for manual intervention to a certain extent, reduces human errors, and makes the process more reliable and stable. In addition, when it is detected that the addition end point has been reached, the system can automatically close the addition container to stop the operation. This closed-loop design ensures the efficiency and accuracy of the addition process, thereby improving the economy and safety of the overall production, and showing innovative advantages that are significantly different from traditional methods. Through this refined monitoring and control mechanism, the present invention not only realizes the intelligence of the addition operation, but also significantly improves the sustainability and resource utilization efficiency of the production process, bringing unexpected industrial value.

[0028] The present invention also discloses an intelligent control song adding system, which includes: a detection module, which is used to detect status information related to the song adding process to obtain data information for analysis; a control module, which is used to calculate the song adding amount according to the acquired data information to generate a song adding strategy; an execution module, which is used to execute corresponding song adding operations based on the generated song adding strategy, wherein the control module can execute the above-mentioned intelligent control song adding method and output corresponding control signals to the execution module to complete the song adding operation.

[0029] According to a preferred embodiment, the execution module includes a fan, a koji adding container, an air supply duct and a closed tank body, wherein one end of the air supply duct is connected to the fan, and the other end thereof is connected to the closed tank body, and the koji adding container connected to the weighing unit configured with the detection module can be connected to the air supply duct in a controllable opening and closing manner, so that the koji powder in the koji adding container can enter the closed tank body through the air supply duct in a controlled manner.

[0030] The intelligent control system for adding koji of the present invention has a multi-level structural design and information feedback mechanism. The detection module configured therein obtains rich data information by real-time monitoring of the state information related to the process of adding koji, and the accumulation of this data provides a solid foundation for subsequent analysis. The control module uses this data information to accurately calculate the amount of koji added, thereby generating a strategy for adding koji that meets actual needs. This process not only improves the accuracy of the operation of adding koji, but also makes the adjustment of the amount of koji added have intelligent characteristics and can adapt to changes under different production conditions. The execution module ensures that the conveying process of koji powder is controllable and efficient by designing the linkage between the koji adding container, the fan and the air supply duct. In this process, the application of the closed tank further enhances the sealing performance of the system, avoids the loss and pollution of koji powder during the transmission process, thereby improving the quality of the product and the safety of the production environment. In particular, the controllable opening and closing design of the koji adding container and the air supply duct makes the entry and exit process of koji powder have a high degree of flexibility and adjustment ability, and can respond and adjust quickly under different production needs. This highly integrated system design not only improves production efficiency, but also significantly reduces the need for manual intervention, thereby reducing human errors and waste of resources. It ultimately forms an intelligent and automated addition operation mode, bringing a new technical perspective and economic value to industrial production, and opening up a new application field for addition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the steps of an intelligent control song adding method according to a preferred embodiment of the present invention;

[0032] Figure 2 It is a flowchart of the steps of the training process of the model algorithm of a preferred embodiment provided by the present invention;

[0033] Figure 3 It is a hardware connection diagram of an intelligent control and music adding system according to a preferred embodiment of the present invention;

[0034] Figure 4 It is a structural schematic diagram of an execution module of a preferred implementation mode provided by the present invention.

[0035] Reference numerals list

[0036] 100: detection module; 110: weighing unit; 200: control module; 300: execution module; 310: fan; 320: bending container; 330: air supply duct; 340: closed tank. DETAILED DESCRIPTION

[0037] The following is a detailed description with reference to the accompanying drawings.

[0038] Example 1

[0039] like Figure 1 As shown, the present invention discloses an intelligent control method for adding a song, which may include one or more of the following steps:

[0040] S1. Front-end detection: detect the status information related to the song adding process to obtain data information for analysis;

[0041] S2, mid-term decision: calculate the amount of songs to be added based on the acquired data information to generate a song adding strategy;

[0042] S3, back-end output: perform corresponding song adding operations based on the generated song adding strategy.

[0043] Preferably, the method for adding quji of the present invention can be applied to mash to which quji powder is to be added, especially mash that is being cooled. The amount of quji added is accurately calculated through an intelligent quji adding algorithm, so that when the corresponding quji adding operation is performed according to the quji adding strategy and the quji adding end point is reached, the quji adding is stopped in time to ensure that the amount of quji powder added meets the usage requirements, wherein meeting the usage requirements means that the deviation between the actual amount of quji powder added and the planned amount of quji powder added is less than a preset threshold.

[0044] Preferably, in the process of brewing liquor, the mash usually needs to be spread out to cool down before adding koji. This is because the temperature of the mash after cooking is usually high, and too high a temperature is not conducive to the growth of yeast and koji powder. Spreading out can quickly reduce the temperature of the mash to a range suitable for fermentation; spreading out can make the moisture distribution of the mash more uniform, ensuring that when adding koji, the moisture content of each part of the mash is consistent, which is conducive to the uniform subsequent fermentation; the mash is easier to mix evenly with the koji powder at an appropriate temperature and humidity, thereby improving the activity of the koji powder and the fermentation effect. Furthermore, the present invention does not limit the spreading and cooling method of the mash, and any method can be used for spreading and cooling, for example, natural spreading, air convection cooling, etc. Further preferably, the mash to be added with koji powder applicable to the present invention can be the mash that is cooled by negative pressure spreading, wherein the negative pressure spreading method refers to using a vacuum pump to suck the closed tank body 340 containing the cooked mash to adjust (mainly reduce) the temperature and humidity of the mash, so that the mash reaches the above-mentioned appropriate temperature and humidity.

[0045] Preferably, the purpose of the front-end detection process of step S1 is to monitor the production environment and material status in real time to ensure that the automatic koji addition operation is performed under optimal conditions. To this end, the status information related to the koji addition process detected in step S1 may include status information related to the production environment and / or status information related to the mash, wherein the status information related to the production environment may include the production environment temperature, the production environment humidity and / or the production environment season; the status information related to the mash may include the mash weight, the mash temperature and / or the mash humidity. Furthermore, in addition to the monitoring of basic parameters such as temperature, humidity, and mash moisture, other key status information may be introduced, such as gas composition analysis, environmental pressure monitoring, and real-time flow detection, etc., to achieve a comprehensive perception of the production environment.

[0046] Preferably, using sensors such as infrared temperature sensors and digital temperature and humidity sensors can ensure accurate temperature and humidity detection of the mash and its surroundings; using capacitive or resistive moisture sensors can efficiently and accurately detect the moisture content of the mash.

[0047] Furthermore, if gas composition analysis is required, gas sensors such as the MQ series sensors can be used. These sensors can monitor the concentrations of gases such as carbon dioxide, carbon monoxide, and volatile organic compounds in the addition environment in real time to ensure that no harmful gases accumulate during the addition process, thereby ensuring the safety and stability of production.

[0048] Preferably, based on the acquired real-time data (mainly the temperature and humidity of the fermented mash), the end point of spreading and airing (i.e., the start point of adding koji) can be determined by comparing it with the set initial values ​​of temperature and humidity, wherein the initial values ​​of temperature and humidity can be set based on the combination of different types of fermented mash and koji powder. Furthermore, when the fermented mash is cooled by negative pressure spreading and airing, when the end point of spreading and airing is reached, the vacuum pump can be turned off to stop the suction, so as to stop spreading and airing and start adding koji.

[0049] For example, in one implementation, the status information shown in the following table may be obtained through step S1:

[0050]

[0051] Preferably, in step S2, the amount of song addition can be accurately analyzed based on the acquired real-time data and the established mid-end decision layer DQN (Deep Q-Network) deep reinforcement learning algorithm to generate a corresponding song addition strategy.

[0052] Preferably, the present invention can use a Q function to represent the expected return of wine production by adding koji under the current production state, wherein the Q function can be used to evaluate the expected return of taking a certain action under a certain state, so as to help the intelligent agent judge the value of choosing a certain action under the current state, and thus make a better decision. Conventional Q functions are usually recorded as Q(s,a) or Q=(s,a), wherein s represents the current state of the environment, and a represents the action taken under this state. The present invention applies the Q function to DQN, which is used to evaluate the value of actions under complex states. In this case, the Q function is a deep neural network, which can process high-dimensional state spaces. Preferably, the Q network of the present invention is as follows:

[0053]

[0054] Among them, s is the condition parameter under the current production environment; temperature is temperature; humidity is humidity; weight_of_dough is the weight of the mash; season is the season of the production environment; a represents the amount of koji added under the condition parameters under the current production environment; θ is the parameter of the Q network, which is used to evaluate the predicted wine production after adding koji powder with the calculated amount of koji under the current production condition parameters; R represents the value range of action a, that is, the action space is a continuous real number domain.

[0055] Preferably, the wine production conditions may be, for example, wine production rate and wine quality.

[0056] Preferably, on the basis of the Q network, the present invention also introduces a Q' network with the same structure as the Q network but different parameters, wherein the structure of the Q' network (i.e., the target network) can be regarded as the same deep neural network as the Q network (i.e., the main network), including the same number of layers and activation functions, but its weights and bias parameters remain fixed during the training process. This design enables the Q' network to provide a relatively stable target Q value for the current state-action pair, thereby helping to reduce instability during the training process. Since the parameters of the Q network are frequently updated, if the main network is used directly to calculate the target Q value, it may cause drastic fluctuations in the Q value estimate, thereby affecting the learning effect of the agent. By introducing the Q' network, the agent can rely on a fixed target network to calculate the target Q value, thereby avoiding noise and uncertainty caused by rapid parameter updates during the learning process. This approach not only improves the stability of the training, but also helps to speed up the convergence speed, so that the agent can learn more effective strategies from the current environmental state and behavior. Preferably, the Q' network of the present invention is as follows:

[0057]

[0058] Among them, θ' is the parameter of the Q' network, which is used to stabilize and guide the learning process of the main network. It is regularly updated from the parameters θ of the main network to maintain the stability of the parameters; τ is the update rate of the entire training process, usually between [0,1], which controls the speed of updating the target network parameters; α is the learning rate, which controls the step size of each parameter update; is the gradient of the main network loss function L with respect to the parameter θ.

[0059] In the above formula, the second line indicates that the parameters θ' of the target network are updated by taking the weighted average of the parameters θ of the current main network and the parameters θ' of the previous target network; the third line indicates that the parameters of the main network are updated by the gradient descent method to minimize the loss function L(θ).

[0060] Preferably, the loss function L(θ) can be calculated using the following formula:

[0061] L(θ)=E[(yQ(s,a;θ)) 2 ]

[0062] Where y is the target value of the loss function, or the target Q value. The loss function L(θ) can be calculated by the variance between the target value of the loss function and the current Q network output.

[0063] Furthermore, the present invention may be provided with a reward function r, which is used to evaluate the wine production rate and wine quality corresponding to the current amount of koji added, and then evaluate its impact. Preferably, the reward function r can be expressed as: r(s, a; s'), wherein s' is a new state, especially a new wine production state, which represents the predicted wine production rate and wine quality after taking the amount of koji added a. If the current amount of koji added can improve the wine production rate (i.e., produce more wine) and wine quality (i.e., the quality of the wine is better), the reward function will give positive feedback, otherwise it will give negative feedback. The goal of the intelligent agent is to maximize the cumulative reward, which means that in the entire production process, the intelligent agent not only pays attention to the current results, but also needs to consider possible future effects. When the intelligent agent chooses the amount of koji added a, it not only needs to consider the current state s, but also needs to evaluate the effect of its decision by predicting the new state s'. The design of the reward function enables the intelligent agent to optimize its koji addition strategy according to the changes in the wine production rate and wine quality, so as to achieve better production results. Therefore, the agent continuously tries different amounts of koji and adjusts its strategy based on the reward information obtained to learn how to choose the optimal amount of koji to improve the overall production effect. This process is a continuous learning and optimization cycle, which ultimately helps the agent achieve the best wine production rate and quality.

[0064] Preferably, when updating the Q network, the Q' network is used to calculate the target Q value, wherein the target Q value is usually calculated based on the maximum Q value of the current state and the next state, and its form is:

[0065] y=r+γmax a′ Q′(s′,a′;θ′)

[0066] Among them, y is the target Q value, which can be calculated by the Bellman equation, providing a recursive way to define the state value function, which represents the wine production reward that can be obtained by the currently selected amount of added koji under a specific state; γ is a discount factor, which is used to achieve a trade-off between immediate rewards and future rewards, reflecting the value of future rewards relative to immediate rewards; a' is the optimal amount of added koji action under the new (wine production) state s', that is, the optimal amount of added koji.

[0067] Preferably, the state-value function reflects the expected value of all possible future returns after taking a certain action in a given state (i.e., performing the addition operation based on the currently selected amount of addition). Through the Bellman equation, the agent can link the value of the current state with the value of the future state, so as to better evaluate the impact of different amounts of addition on the wine production revenue. In other words, the Bellman equation provides a systematic method that enables the agent to gradually update and optimize its state-value function based on the current amount of addition and possible future results, and ultimately achieve the best wine production strategy. This recursive relationship helps the agent to learn and make decisions effectively in a complex decision-making process, thereby improving the overall efficiency of wine production. Preferably, the discount factor affects the agent's evaluation of the current addition strategy, and a higher γ value means that this addition strategy will be considered more valuable in the future, encouraging long-term execution.

[0068] Preferably, the present invention may adopt the following greedy strategy to balance the behaviors of exploration and exploitation in the song adding strategy:

[0069]

[0070] Here, ε is the exploration rate.

[0071] Preferably, the present invention may be provided with an experience replay buffer buffer = {(s, a, r, s')}, so as to extract samples from the experience replay buffer for training. The experience replay buffer is an important component in reinforcement learning, which stores the tuple of state, action, reward and next state generated by the agent's interaction with the environment. This mechanism allows the agent to randomly extract historical experience for learning during training, rather than relying solely on recent experience, thereby effectively reducing correlation and improving learning efficiency.

[0072] Preferably, if Figure 2 As shown, in step S2, the training process of the model algorithm may include one or more of the following steps:

[0073] S2.1, initialize Q network and Q' network;

[0074] S2.2, create an experience replay buffer;

[0075] S2.3, collect test data;

[0076] S2.4, select the action of adding bending amount;

[0077] S2.5, storing the experience in the experience replay buffer;

[0078] S2.6, randomly sample experience from the experience replay buffer;

[0079] S2.7, calculate the target Q value;

[0080] S2.8, calculate the prediction value of the current Q network for the sampled sample, and update the Q network;

[0081] S2.9, reduce the exploration rate and repeat the training;

[0082] S2.10. Training is completed and the model is evaluated.

[0083] Preferably, in step S2.1, two Q-value networks may be defined first: one is the main network (i.e., the Q network), which is used for learning and updating; the other is the target network (i.e., the Q' network), which is used to stabilize the training process. Preferably, the Q network is usually initialized to zero or a small random value. Choosing a zero value can make the agent more conservative in the early stages of learning and avoid premature use of insufficient information. The initial weights of the Q' network can be the same as those of the Q network to ensure that their outputs are consistent at the beginning of training. The weights of the target network are updated regularly with the training cycle to ensure the stability of the learning process and reduce the oscillation of the Q value. Preferably, the Q network is usually constructed using a deep neural network, receiving state information at the input layer, the output layer corresponding to all possible curvature actions, and the hidden layer responsible for the abstract representation of features.

[0084] Preferably, in step S2.2, the experience replay buffer can be designed as a fixed-size FIFO (first-in-first-out) queue. When the experience replay buffer is full, the oldest experience will be replaced by the new experience. Furthermore, the agent can store each experience (state, action, reward, and next state) in the experience replay buffer so that a small batch (mini-batch) of historical experience can be randomly extracted during subsequent training. By learning from multiple states and actions, the agent can better generalize to new, unseen states. In addition, the experience replay buffer allows the agent to use richer samples when updating, thereby improving learning efficiency and stability.

[0085] Preferably, in step S2.3, the agent may collect the state information of the current production environment obtained in step S1, which generally includes various factors affecting the amount of koji added, such as humidity, temperature, mash weight and season. Further, the state information of the current production environment may be combined into a state vector for subsequent input into the Q network for learning. Preferably, the collected data may be standardized or normalized for input into the neural network, thereby ensuring that different features have similar scales, which may improve the effectiveness of training.

[0086] Preferably, in step S2.4, the agent may use an ε-greedy strategy to select a curvature action according to the current state, wherein the strategy may find a balance between exploration and utilization so that the agent can learn the optimal strategy. Furthermore, when using the ε-greedy strategy, the agent may select the curvature action currently considered optimal in most cases, but may occasionally randomly select a curvature action for exploration, wherein the probability of randomly selecting a curvature action is the exploration rate ε; and the probability of selecting the optimal curvature action is 1-ε. Preferably, randomly selecting a curvature action helps to explore new strategies and states, thereby avoiding falling into a local optimal solution, and by selecting the curvature action with the largest Q value in the current Q network, the agent can use existing experience to maximize the expected reward.

[0087] Preferably, in step S2.5, after successfully selecting a song-adding action, the agent can store the state information of the current production environment, the selected song-adding action, the immediate reward and the new (wine production) state into the experience playback buffer to ensure that the agent can use these historical experiences for updating in the subsequent learning process.

[0088] Preferably, in step S2.6, the agent can periodically randomly extract a batch of experiences from the experience playback buffer for learning and updating, thereby effectively breaking the correlation between experiences and improving the generalization ability of the model. Through batch learning, the agent can use multiple experiences in one update to improve the efficiency of learning. The introduction of randomly sampled experience solves the correlation problem in the learning process and avoids overfitting of the model due to reliance on the latest experience during training. By learning from a variety of different experiences, the agent can more comprehensively understand the value of state-action pairs, thereby promoting generalization. Batch learning not only improves computational efficiency, but also reduces noise in the training process, which helps the model to converge stably to the optimal strategy.

[0089] Preferably, in step S2.7, after sampling the experience, the agent can calculate the target Q value according to the Bellman equation, wherein the target Q value is used to evaluate the expected return that can be obtained by selecting a certain curvature action in a specific state, and provide a benchmark for updating the Q network. The target Q value is the goal that the agent hopes to learn, and the output of the Q network should be as close to this value as possible. The target Q value helps the agent evaluate the quality of its behavior strategy by combining immediate rewards and future expected returns. The Bellman equation provides a recursive method for the agent to estimate the long-term returns of curvature actions, ensuring that future impacts are taken into account during the learning process. By continuously updating the Q value, the agent can gradually optimize its strategy, thereby achieving better decision-making in complex environments.

[0090] Preferably, in step S2.8, the agent can calculate the predicted value of the current Q value network for the sampled sample after calculating the target Q value, so as to evaluate the difference between the current strategy and the target strategy, thereby guiding the subsequent update. Further, based on the current state and the curvature action, the agent can calculate the current Q value through the Q network, wherein the difference between the current Q value and the target Q value will be used to calculate the loss, thereby updating the parameters of the Q network. The current Q value reflects the agent's evaluation of a curvature action based on existing experience, while the target Q value is the goal that the agent strives to pursue. By comparing the difference between the two, the agent can identify areas that need improvement, thereby gradually optimizing its strategy during the training process. This process ensures that the agent can flexibly adapt and make corresponding adjustments in a constantly updated environment. Further, after calculating the current Q value prediction and the target Q value, the agent can use the minimization loss function to update the parameters of the Q network to reduce the difference between the predicted value and the true value. Preferably, the agent can update the parameters of the Q network by using a gradient descent method or a variant thereof (such as an Adam optimizer). By minimizing the loss function, the agent can gradually approach the target Q value, thereby optimizing the decision-making strategy. This process ensures the adaptability of the model to environmental feedback, enabling the agent to continuously improve its decision-making ability in a complex and dynamic environment. Effective parameter updates not only help improve the accuracy of the model, but also speed up its convergence, allowing the agent to reach the optimal strategy as quickly as possible.

[0091] Preferably, in step S2.9, the agent may reduce the exploration rate ε so as to make more use of existing knowledge in the later stage of learning. By gradually reducing exploration, the agent can more robustly execute the learned strategy. Preferably, the agent can set an initial exploration rate and a final exploration rate, as well as a decay factor, and adjust the exploration rate according to the decay factor after each training cycle, so as to achieve repeated training. Furthermore, a high exploration rate can be maintained in the early stage of training to ensure that the agent can widely explore different strategies; as learning deepens, the exploration rate can be gradually reduced to increase the utilization of learned knowledge. A high exploration rate enables the agent to obtain rich data in the early stage of training, helping it to understand the environment and optimize strategies. In the later stage of training, reducing the exploration rate can reduce the randomness of the model in a known state, ensuring that the agent can stably execute the optimal strategy. This process is designed to balance the relationship between exploration and utilization, so that the agent can make full use of existing experience in the learning process, while constantly seeking new room for improvement.

[0092] Preferably, in step S2.10, the agent can evaluate the trained model algorithm on the test set to verify its reliability and stability under different environmental conditions. Preferably, a variety of evaluation indicators are designed during the evaluation, such as accuracy, stability, convergence speed, etc., to comprehensively evaluate the model performance, and these indicators can be calculated by comparing the difference between the model output and the actual output. Furthermore, cross-validation can be used to ensure the consistency of the model's performance on different data sets to avoid overfitting. Furthermore, the model algorithm is tested on a small scale in a real production environment to observe its optimization effect on the amount of added curves, and the model algorithm is further adjusted according to the test results.

[0093] Preferably, once the model has been evaluated and meets the expected performance criteria, the agent's strategy can be deployed in the actual brewing process. The established model algorithm can accurately calculate the optimal amount of koji to add, thereby generating a koji adding strategy.

[0094] Exemplarily, in one embodiment, state s is as follows: the production environment temperature is 15°C, the production environment humidity is 70%, the production environment season is spring, the mash weight is 500kg, the mash temperature is 35°C, and the mash humidity is 55%. On this basis, the amount of koji added to action a is set to 50kg, 55kg, 60kg, 65kg, and 70kg. After calculation in step S2, the Q values ​​of the action a corresponding to the above five different amounts of koji added in the Q network output are: 10.5, 11.0, 12.5, 11.8, and 10.0, respectively. Therefore, the Q value corresponding to the amount of koji added of 60kg is the highest, which is 12.5, which means that in the current state, choosing the amount of koji added of 60kg will bring the best wine production rate and wine quality. In other words, the agent can choose 60kg as the amount of koji added in this state in the hope of obtaining the highest return. Further, according to the greedy strategy, if you choose to use, the agent will choose 60kg; if you choose to explore, the agent will try other amounts of koji added. After executing the action, the Q network is updated according to the actual reward obtained to continuously optimize the song adding strategy.

[0095] Preferably, in step S3, the koji adding operation is performed according to the koji adding strategy generated in step S2, wherein the change in the koji powder weight in the koji adding container 320 can be monitored by weighing to determine the koji adding end point. Preferably, when the actual amount of koji added reaches the calculated optimal amount of koji added, that is, the koji adding end point is reached, the koji adding container 320 can be closed to stop the koji adding operation.

[0096] Example 2

[0097] This embodiment is a further improvement on Embodiment 1, and the repeated contents will not be repeated here.

[0098] like Figure 3As shown, the present invention also discloses an intelligent control song adding system, which includes a detection module 100, a control module 200 and an execution module 300, wherein the control module 200 is capable of executing the intelligent control song adding method as described in Example 1, and outputting a corresponding control signal to the execution module 300 to complete the song adding operation.

[0099] Preferably, the detection module 100 may include various sensors, such as a temperature sensor, a humidity sensor, etc., to perform the front-end detection process described in step S1 of embodiment 1 and obtain corresponding data information. Furthermore, the data information obtained by the detection module 100 may be sent to the control module 200, so that the control module 200 can calculate the optimal amount of adding music according to the data information obtained by the detection module 100, thereby driving the execution module 300 to perform the corresponding adding music operation according to the adding music strategy attached to the control signal.

[0100] Preferably, the detection module 100 may also include a weighing unit 110 for acquiring weight data, so that when the execution module 300 performs the corresponding addition operation according to the addition strategy, the control module 200 may determine the end point of the addition according to the weight data acquired in real time by the weighing unit 110.

[0101] Preferably, if Figure 4 As shown, the execution module 300 may include a fan 310, a koji container 320, an air supply duct 330 and a closed tank 340, wherein one end of the air supply duct 330 may be connected to the fan 310, and the other end thereof may be connected to the closed tank 340, and the koji container 320 may be connected to the air supply duct 330 in a controllable opening and closing manner, so that the koji powder in the koji container 320 may enter the closed tank 340 through the air supply duct 330 in a controlled manner. Preferably, the weighing unit 110 may be connected to the koji container 320, and may be particularly arranged below the koji container 320 to monitor the weight change of the koji container 320, so as to calculate the actual koji addition amount in real time. Further, when the actual koji addition amount reaches the optimal koji addition amount calculated by the control module 200, the control module 200 may close the koji container 320 to stop the koji addition operation.

[0102] Preferably, the execution module 300 may be provided with a locking unit on the closed tank body 340, so that after adding mash into the closed tank body 340, the cover body is locked by the locking unit to ensure that the entire koji adding process is completely in a closed state, effectively solving the problem of koji powder dust overflowing during the conventional koji adding process.

[0103] Preferably, the execution module 300 may further include a flipping unit and / or a stirring unit, wherein the above two functional units may be independently provided or simultaneously provided to achieve mixing of the koji powder and the material in the closed tank body 340 .

[0104] In one embodiment, the control module 200 can be set to start the qu addition program when the temperature of the fermented grains drops to 18-22°C after airing. The detection module 100 can send the acquired data information such as the temperature and humidity of the production environment to the control module 200, so that the control module 200 can use the established model algorithm to accurately analyze the amount of qu addition, thereby outputting the optimal amount of qu addition and generating a corresponding qu addition strategy. The execution module 300 can perform the corresponding qu addition operation according to the qu addition strategy, and close the qu addition container 320 under the control of the control signal generated by the control module 200 when reaching the end of qu addition. Preferably, when the end of qu addition is about to be reached, the execution module 300 controlled by the control signal generated by the control module 200 can reduce the outlet of the qu addition container 320 to achieve a micro-adjustment of the final qu addition process. The intelligent control qu addition system of the present invention can accurately control the amount of qu addition to 0.1%.

[0105] It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention specification and its drawings are illustrative and do not constitute a limitation of the claims. The scope of protection of the present invention is defined by the claims and their equivalents. The present invention specification contains multiple inventive concepts, such as "preferably" or "according to a preferred embodiment", which means that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application based on each inventive concept. Throughout the text, the features guided by "preferably" are only an optional method and should not be understood as a must-have setting. Therefore, the applicant reserves the right to abandon or delete the relevant preferred features at any time.

Claims

1. An intelligent control method for adding music, characterized in that: It includes: Front-end detection: detect the status information related to the song adding process to obtain data information for analysis; Mid-range decision-making: Calculate the amount of music to be added based on the acquired data information to generate a music addition strategy; Backend output: Execute the corresponding song adding operation based on the generated song adding strategy. Among them, in the mid-end decision-making process, a DQN deep reinforcement learning algorithm is established and trained to evaluate the potential impact of different amounts of added koji on wine production rate and quality under the current production status obtained in the front-end detection process, so as to determine the expected return of taking a specific amount of added koji action, thereby determining the optimal amount of added koji and generating a koji addition strategy for execution by the back-end output process.

2. The intelligent control method for adding music according to claim 1, characterized in that: In the mid-level decision-making process, a Q network is defined and trained to approximate a Q function to quantify the expected reward of taking a specific additive action under a specific production state, and then the Q' network is introduced to stabilize the learning process. The state information of the current production state can be merged into a state vector for subsequent input into the Q network for learning.

3. The intelligent control method for adding music according to claim 1 or 2, characterized in that: The status information of the current production status includes status information related to the production environment and / or status information related to the mash, wherein the status information related to the production environment includes the production environment temperature, the production environment humidity and / or the production environment season; the status information related to the mash includes the mash weight, the mash temperature and / or the mash humidity, and the acquired status information can be input into the Q network after being standardized or normalized.

4. The intelligent control method for adding music according to any one of claims 1 to 3, characterized in that: An ε-greedy strategy is used to select a bending action according to the current state, wherein the bending action is randomly selected with a probability based on the exploration rate ε or the optimal bending action is selected with a probability of 1-ε; and the experience corresponding to the selected bending action is stored in the experience playback buffer, wherein the experience includes the state information of the current production environment, the selected bending action, the immediate reward and the new state.

5. The intelligent control method for adding music according to any one of claims 1 to 4, characterized in that: A batch of experiences are randomly sampled from the experience replay buffer for learning and updating at regular intervals. After sampling the experiences, the target Q value is calculated according to the Bellman equation. The target Q value is used to evaluate the expected return that can be obtained by selecting a certain curvature action in a specific state, and provides a benchmark for updating the Q network.

6. The intelligent control method for adding music according to any one of claims 1 to 5, characterized in that: Based on the current state and the added curvature action, the current Q value is calculated through the Q network, and the loss is calculated by minimizing the loss function according to the difference between the current Q value and the target Q value, so as to update the parameters of the Q network by the gradient descent method or its variants.

7. The intelligent control method for adding music according to any one of claims 1 to 6, characterized in that: By setting an initial exploration rate and a final exploration rate, as well as a decay factor, and adjusting the exploration rate according to the decay factor after each training cycle, the exploration is gradually reduced until the training is completed. The training set is used to evaluate the trained model algorithm to verify its reliability and stability under different environmental conditions.

8. The intelligent control method for adding music according to any one of claims 1 to 7, characterized in that: During the back-end output process, the corresponding addition operation is performed according to the generated addition strategy, and the change in the weight of the koji powder in the addition container (320) is monitored by weighing to determine the end point of addition. When the end point of addition is reached, the addition container (320) is closed to stop the addition operation.

9. An intelligent control song adding system, characterized in that: It includes: A detection module (100) is used to detect status information related to the song adding process to obtain data information for analysis; A control module (200) is used to calculate the amount of song addition according to the acquired data information to generate a song addition strategy; An execution module (300) is used to execute a corresponding song adding operation based on the generated song adding strategy, The control module (200) is capable of executing the intelligent control method for adding a song as claimed in any one of claims 1 to 8, and outputting a corresponding control signal to the execution module (300) to complete the operation of adding a song.

10. The intelligent control music adding system according to claim 9, characterized in that: The execution module (300) includes a fan (310), a koji container (320), an air supply duct (330) and a closed tank body (340), wherein one end of the air supply duct (330) is connected to the fan (310), and the other end thereof is connected to the closed tank body (340), and the koji container (320) connected to the weighing unit (110) configured with the detection module (100) can be connected to the air supply duct (330) in a controllable opening and closing manner, so that the koji powder in the koji container (320) can enter the closed tank body (340) through the air supply duct (330) in a controlled manner.

Images