Intelligent control song adding method and system

Abstract

The present invention relates to an intelligent control method and system for adding koji. The method comprises: front-end detection: detecting status information related to the koji addition process to obtain data information for analysis; mid-end decision-making: calculating the amount of koji added based on the obtained data information to generate a koji addition strategy; back-end output: executing corresponding koji addition operations based on the generated koji addition strategy. In the mid-end decision-making process, a DQN deep reinforcement learning algorithm is established and trained to evaluate the potential impact of different koji addition amounts on wine production rate and quality under the current production status obtained in the front-end detection process, so as to determine the expected return of taking a specific koji addition action, thereby determining the optimal koji addition amount and generating a koji addition strategy for execution in the back-end output process. The system comprises a detection module, a control module, and an execution module, wherein the control module is capable of executing the above-mentioned method and outputting corresponding control signals to the execution module to complete the koji addition operation.

Description

Technical Field

[0001] The present invention relates to the technical field of automated liquor brewing, and in particular to an intelligent control method and system for adding koji. Background Art

[0002] Adding koji (dàqu) is a key step in the baijiu (white spirit) brewing process. As a unique saccharifying and fermenting agent, daqu (daqu) contains a rich variety of bacterial, biological, and enzyme systems. It plays a role in saccharification, fermentation, and aroma development during the baijiu production process, giving the liquor its unique flavor. In traditional baijiu brewing, a certain amount of koji powder is added to the mash after it has been aired and cooled. The amount of koji powder is dynamically adjusted based on factors such as the current ambient temperature, and the amount must be precisely controlled to ensure the fermentation quality of the mash.

[0003] CN114480048A discloses a wine lees spreading and drying system and koji adding system, which includes a feeding device, a spreading device, a turning device, a koji adding device and a mixing and discharging device; the spreading device includes a chain conveyor belt, an air chamber and a fan; the air chamber is arranged below the chain conveyor belt, the air chamber is provided with an air inlet and an air outlet, the fan is connected to the air inlet of the air chamber, and the chain conveyor belt is provided with ventilation holes; the feeding device is arranged at the feeding end of the chain conveyor belt, for adding wine lees to the chain conveyor belt; the turning device is arranged above the chain conveyor belt, for turning over the wine lees on the chain conveyor belt; the koji adding device is arranged above the chain conveyor belt, for adding koji to the wine lees on the chain conveyor belt; the mixing and discharging device is arranged at the discharging end of the chain conveyor belt, for stirring the wine lees and koji, and sending the stirred wine lees and koji to the target position.

[0004] CN106434233A discloses an intelligent control system for liquor production, including a control device, a robotic retort loading system, a PLC, various pumps, valves, motors, sensors, and instruments. Furthermore, a method for intelligent liquor production using this intelligent control system is disclosed, including steps such as weighing ingredients, automatically loading the retort, distilling liquor, cooling, and adding koji.

[0005] Existing koji addition technology primarily uses volumetric metering to control the amount of koji powder added. However, this method presents several key issues. First, due to the difference in the flow rates of the mash and koji powder, their addition processes cannot be synchronized. Specifically, when the mash is finished drying and the koji powder is added, there may still be residual koji powder in the koji addition bucket; conversely, before the mash is finished drying and the koji is added, the koji powder in the koji addition bucket may have been used up. This asynchronous addition method prevents the effective synchronous compound addition of materials, which may affect the final fermentation effect and wine quality. In addition, the koji addition operation is usually carried out in an open environment, which can cause the koji powder to be released into the air during the addition process, causing pollution to the production environment and waste of resources. These problems urgently need to be solved to improve the efficiency of liquor brewing and improve product quality.

[0006] In addition, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making the present invention, but due to space limitations, not all details and contents are listed in detail. However, this does not mean that the present invention does not have the characteristics of these prior arts. On the contrary, the present invention already has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art to the background technology. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention provides an intelligent control method and system for adding music to solve at least part of the above technical problems.

[0008] The present invention discloses an intelligent control method for adding music, which comprises:

[0009] Front-end detection: Detect the status information related to the song addition process to obtain data information for analysis;

[0010] Mid-end decision-making: Calculate the number of songs to be added based on the acquired data to generate a song addition strategy;

[0011] Backend output: Execute corresponding song adding operations based on the generated song adding strategy.

[0012] Preferably, in the mid-end decision-making process, a DQN deep reinforcement learning algorithm is established and trained to evaluate the potential impact of different amounts of added koji on wine production rate and wine quality under the current production status obtained in the front-end detection process, so as to determine the expected return of taking a specific amount of added koji action, thereby determining the optimal amount of added koji and generating a koji addition strategy for execution by the back-end output process.

[0013] The intelligent control method for adding koji of the present invention completes the process of adding koji through front-end detection, mid-end decision-making and back-end output. First, the front-end detection stage monitors the status information related to the addition of koji in real time, so that the system can obtain accurate data and significantly improve its sensitivity to raw materials and production conditions. Then, in the mid-end decision-making stage, the deep reinforcement learning algorithm (DQN) is used to analyze these data to evaluate the potential impact of different amounts of koji added on wine production rate and wine quality, thereby forming a dynamic koji addition strategy and reducing the error of empirical decision-making. Finally, in the back-end output stage, accurate koji powder addition is performed according to the generated optimal koji addition strategy to ensure that the optimal amount of koji added under specific production conditions is implemented. Through the synergistic effect of the above three levels, the present invention realizes a closed-loop feedback mechanism from data to decision-making to execution, so that each step in the process of adding koji can be adjusted based on the optimal strategy, ultimately significantly improving production efficiency and product quality. The present invention not only has a breakthrough significance in the traditional winemaking industry, but also provides a systematic solution that can be used as a reference in the field of intelligent manufacturing.

[0014] According to a preferred embodiment, in the mid-level decision-making process, a Q network is defined and trained to approximate a Q function to quantify the expected reward of taking a specific additive action under a specific production state, and then a Q' network is introduced to stabilize the learning process, wherein the state information of the current production state can be merged into a state vector for subsequent input into the Q network for learning.

[0015] By defining a Q network and training it into an approximate Q function, the present invention can quantify the expected returns of taking different actions of adding koji under specific production conditions. This quantification not only provides a clear assessment of the expected effect, but also provides a reliable basis for subsequent decision-making, avoiding the blindness and uncertainty of traditional rules of thumb. Furthermore, the introduction of the Q' network strategy makes the learning process more stable and reduces fluctuations caused by overfitting. This mechanism ensures the adaptability of the model under different production conditions, allowing the system to respond quickly to environmental changes. In addition, by merging the state information of the current production state into a state vector, the Q network can efficiently process multi-dimensional data, thereby improving learning efficiency and accuracy. In summary, this layer-by-layer progressive technical feature not only improves the intelligence level of the koji addition process, but also provides a new perspective for precise control and optimization of production, significantly enhancing the ability to control quality and output during the winemaking process.

[0016] According to a preferred embodiment, the status information of the current production status includes status information related to the production environment and / or status information related to the mash, wherein the status information related to the production environment includes the production environment temperature, the production environment humidity and / or the production environment season; the status information related to the mash includes the mash weight, the mash temperature and / or the mash humidity, and the acquired status information can be input into the Q network after being standardized or normalized.

[0017] The current state information for the production process includes multiple indicators related to the production environment and mash, such as the temperature, humidity, and season of the production environment, as well as the weight, temperature, and humidity of the mash. This comprehensive data collection ensures that the intelligent agent fully understands changes in production conditions, providing rich context for subsequent decision-making. Furthermore, this standardized or normalized state information is input into the Q-network, enabling unified processing of data of varying dimensions and ranges, improving the model's sensitivity and adaptability to various input signals. This data preprocessing method not only reduces noise in the input data but also accelerates convergence during learning and enhances model stability. Consequently, the Q-network can more accurately estimate the expected rewards of different koji addition actions under specific environmental conditions, enabling more refined control and optimization. Ultimately, this combination of correlated features and processing methods significantly enhances the intelligence of the winemaking process, opening up a new path for optimizing product quality and production efficiency, surpassing the capabilities of traditional methods.

[0018] According to a preferred embodiment, an ε-greedy strategy is used to select a quantification action according to the current state, wherein the quantification action is randomly selected with a probability based on the exploration rate ε or the optimal quantification action is selected with a probability of 1-ε; and the experience corresponding to the selected quantification action is stored in the experience playback buffer, wherein the experience includes the state information of the current production environment, the selected quantification action, the immediate reward and the new state.

[0019] This invention employs a mechanism based on an exploration rate ε, enabling the intelligent agent to balance exploration and exploitation when selecting additive actions. When the exploration rate ε is set to an appropriate value, the system not only selects the currently known optimal action with a probability of 1-ε, thereby accelerating the learning process and maximizing benefits, but also randomly selects additive actions with a probability of ε to explore new possibilities. This exploration mechanism prevents the system from being trapped in local optima and enhances its ability to fully explore the state space. Furthermore, the selected action, along with the corresponding current state information, immediate reward, and new state, is stored in an experience replay buffer. This design allows the system to reuse this valuable experience in subsequent learning, deepening its understanding of the relationship between specific production states and additive responses. In this way, the system achieves dynamic learning capabilities, continuously optimizes decision-making strategies, and rapidly adapts to environmental changes. Ultimately, this series of interrelated feature combinations not only enhances the intelligence of the additive process but also provides a solid foundation for optimizing production efficiency and improving product quality. It transcends the limitations of traditional decision-making methods and opens up a more efficient and flexible production model.

[0020] According to a preferred embodiment, a batch of experiences are randomly sampled from the experience replay buffer for learning and updating. After sampling the experiences, the target Q value is calculated according to the Bellman equation. The target Q value is used to evaluate the expected reward that can be obtained by selecting a certain action to increase the amount of distortion in a specific state, and provides a benchmark for updating the Q network.

[0021] The random sampling mechanism enables the system to learn from diverse experiences, avoiding overfitting caused by temporal data correlation. This mechanism ensures data independence during training and enhances the model's adaptability to environmental changes. By calculating the target Q-value using the Bellman equation, the system accurately estimates the expected reward of selecting a specific curvature action in a specific state. This not only provides a quantitative benchmark for Q-network updates but also fosters a deeper understanding of future outcomes. The introduction of target Q-values ​​enhances the directionality of the learning process, clarifies the model's optimization objectives in different states, and provides a scientific basis for policy improvement. This method of extracting target rewards from experience makes the learning process more efficient and targeted, thereby accelerating the convergence of the Q-network and improving decision quality. Ultimately, this series of progressive feature integration not only significantly improves the intelligence level of curvature action selection but also lays a solid foundation for dynamic optimization of the production process, enabling the system to demonstrate unprecedented flexibility and accuracy in complex and changing production environments.

[0022] According to a preferred embodiment, based on the current state and the added amount action, the current Q value is calculated by the Q network, so as to calculate the loss by minimizing the loss function according to the difference between the current Q value and the target Q value, thereby updating the parameters of the Q network by the gradient descent method or its variants.

[0023] This process uses a Q-network to map states and action variables to corresponding Q-values, enabling real-time reflection of the system's decision value in specific states. This dynamic evaluation mechanism provides strong support for policy optimization. By calculating the difference between the current Q-value and the target Q-value, the system effectively identifies shortcomings in the current policy. This difference serves as the basis for a loss function, enabling precise feedback adjustments. This policy adjustment not only relies on past experience but also dynamically adapts to current environmental changes, demonstrating the adaptive nature of the learning process. Using gradient descent or its variants to update the Q-network parameters makes this adjustment process more efficient and stable, ensuring that the model convergence is not affected by noise or local extrema. By constructing this series of closely related features, the system not only improves the accuracy and intelligence of decisions but also significantly accelerates the convergence of the learning process. This enables more efficient resource allocation and optimization in complex production environments, surpassing the expectations of traditional optimization methods and demonstrating new potential for intelligent decision-making.

[0024] According to a preferred embodiment, an initial exploration rate and a final exploration rate, as well as a decay factor, are set, and the exploration rate is adjusted according to the decay factor after each training cycle, thereby gradually reducing the exploration until the training is completed. The trained model algorithm is evaluated using the training set to verify its reliability and stability under different environmental conditions.

[0025] The setting of the initial exploration rate enables the model to conduct extensive exploration in the early stages of training and obtain rich empirical data. This process provides a diverse background and rich strategic choices for model learning. As training progresses, the exploration rate is gradually reduced through the attenuation factor, so that the model can focus on using existing knowledge to optimize decisions in the later stages. This strategy of gradually reducing exploration ensures that the model can be more stably refined in the final stage without performance fluctuations due to excessive exploration. At the same time, the trained model is evaluated using the training set to further verify the reliability and stability of the model under different environmental conditions, ensuring that it can maintain consistent performance in real applications. This evaluation process not only provides quantitative verification of the model's generalization ability, but also provides a basis for subsequent model improvements and parameter adjustments, thereby forming a closed-loop optimization mechanism. Ultimately, through the combination of this dynamic adjustment of the exploration rate and model evaluation, the present invention can significantly improve the model's adaptability and decision-making accuracy in complex environments, surpassing the limitations of traditional methods and demonstrating a higher level of intelligence and long-term effectiveness.

[0026] According to a preferred embodiment, during the back-end output process, the corresponding koji adding operation is performed according to the generated koji adding strategy, and the change in the koji powder weight in the koji adding container is monitored by weighing to determine the koji adding end point, wherein when the koji adding end point is reached, the koji adding container is closed to stop the koji adding operation.

[0027] The present invention implements the corresponding addition operation through the generated addition strategy, ensuring the accuracy and consistency of the operation, so that each addition process can strictly follow the preset parameters, thereby optimizing production efficiency. The introduction of the weighing method provides an effective means for real-time monitoring of the changes in the weight of the koji powder in the addition container. This dynamic feedback mechanism enables the system to obtain key data in the addition process in a timely manner, and then judge whether the addition reaches the predetermined end point. This judgment method based on weight change not only improves the degree of automation of the addition operation, but also reduces the need for manual intervention to a certain extent, reduces human error, and makes the process more reliable and stable. In addition, when it is detected that the addition end point has been reached, the system can automatically close the addition container to stop the operation. This closed-loop design ensures the efficiency and accuracy of the addition process, thereby improving the economy and safety of the overall production, and showing innovative advantages that are significantly different from traditional methods. Through this refined monitoring and control mechanism, the present invention not only realizes the intelligence of the addition operation, but also significantly improves the sustainability and resource utilization efficiency of the production process, bringing unexpected industrial value.

[0028] The present invention also discloses an intelligent control song addition system, which includes: a detection module, which is used to detect status information related to the song addition process to obtain data information for analysis; a control module, which is used to calculate the song addition amount based on the acquired data information to generate a song addition strategy; an execution module, which is used to execute corresponding song addition operations based on the generated song addition strategy, wherein the control module can execute the above-mentioned intelligent control song addition method and output corresponding control signals to the execution module to complete the song addition operation.

[0029] According to a preferred embodiment, the execution module includes a fan, a koji adding container, an air supply duct and a closed tank body, wherein one end of the air supply duct is connected to the fan, and the other end thereof is connected to the closed tank body, and the koji adding container connected to the weighing unit configured with the detection module can be connected to the air supply duct in a controllable opening and closing manner, so that the koji powder in the koji adding container can enter the closed tank body through the air supply duct in a controlled manner.

[0030] The intelligent control system for adding koji powder of the present invention has a multi-level structural design and information feedback mechanism. The detection module configured therein obtains rich data information by real-time monitoring of the state information related to the adding koji powder process. The accumulation of this data provides a solid foundation for subsequent analysis. The control module uses this data information to accurately calculate the amount of koji powder to be added, thereby generating a koji powder adding strategy that meets actual needs. This process not only improves the accuracy of the adding koji powder operation, but also makes the adjustment of the amount of koji powder have intelligent features and can adapt to changes under different production conditions. The execution module ensures that the conveying process of koji powder is controllable and efficient by designing the linkage between the adding koji powder container, the blower and the air supply duct. In this process, the application of the closed tank further enhances the sealing performance of the system, avoids the loss and pollution of koji powder during the transmission process, thereby improving the quality of the product and the safety of the production environment. In particular, the controllable opening and closing design of the adding koji powder container and the air supply duct makes the entry and exit process of the koji powder have a high degree of flexibility and adjustment ability, and can respond and adjust quickly under different production needs. This highly integrated system design not only improves production efficiency, but also significantly reduces the need for manual intervention, thereby reducing human errors and waste of resources. Ultimately, it forms an intelligent and automated addition operation mode, bringing a new technical perspective and economic value to industrial production, and creating a new application field for addition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of the steps of the intelligent control song adding method according to a preferred embodiment of the present invention;

[0032] Figure 2 This is a flowchart of the steps of the training process of the model algorithm of a preferred embodiment provided by the present invention;

[0033] Figure 3 This is a hardware connection diagram of an intelligent control and music adding system according to a preferred embodiment of the present invention;

[0034] Figure 4 It is a structural diagram of an execution module of a preferred embodiment provided by the present invention.

[0035] Reference Signs List

[0036] 100: Detection module; 110: Weighing unit; 200: Control module; 300: Execution module; 310: Fan; 320: Curing container; 330: Air supply duct; 340: Sealed tank. DETAILED DESCRIPTION

[0037] The following is a detailed description with reference to the accompanying drawings.

[0038] Example 1

[0039] like Figure 1 As shown, the present invention discloses an intelligent control method for adding songs, which may include one or more of the following steps:

[0040] S1. Front-end detection: Detect the status information related to the song addition process to obtain data information for analysis;

[0041] S2, mid-end decision-making: Calculate the number of songs to be added based on the acquired data information to generate a song addition strategy;

[0042] S3, backend output: Execute corresponding song adding operations based on the generated song adding strategy.

[0043] Preferably, the method for adding koji of the present invention is applicable to mash to which koji powder is to be added, especially mash that is undergoing cooling. The amount of koji added is accurately calculated by an intelligent koji addition algorithm, so that when the corresponding koji addition operation is performed according to the koji addition strategy and the koji addition end point is reached, the koji addition is stopped in time to ensure that the amount of koji powder added meets the usage requirements, wherein meeting the usage requirements means that the deviation between the actual amount of koji powder added and the planned amount of koji powder added is less than a preset threshold.

[0044] Preferably, during the liquor brewing process, the mash is usually spread out to cool before adding koji. This is because the temperature of the mash after cooking is usually high, and excessively high temperatures are not conducive to the growth of yeast and koji powder. Spreading out can quickly reduce the temperature of the mash to a range suitable for fermentation. Spreading out can make the moisture distribution of the mash more uniform, ensuring that the moisture content of each part of the mash is consistent when adding koji, which is conducive to the uniform progress of subsequent fermentation. The mash is easier to mix evenly with the koji powder at an appropriate temperature and humidity, thereby improving the activity of the koji powder and the fermentation effect. Furthermore, the present invention does not limit the method of spreading out to cool the mash, and any method can be used for spreading out to cool, for example, natural spreading, air convection cooling, etc. Further preferably, the mash to be added with koji powder applicable to the present invention can be the mash that is cooled by negative pressure spreading, wherein the negative pressure spreading method refers to using a vacuum pump to suck the closed tank body 340 containing the cooked mash to adjust (mainly reduce) the temperature and humidity of the mash, so that the mash reaches the above-mentioned appropriate temperature and humidity.

[0045] Preferably, the purpose of the front-end detection process of step S1 is to monitor the production environment and material status in real time to ensure that the automatic koji addition operation is performed under optimal conditions. To this end, the status information related to the koji addition process detected in step S1 may include status information related to the production environment and / or status information related to the mash, wherein the status information related to the production environment may include the production environment temperature, the production environment humidity and / or the production environment season; the status information related to the mash may include the mash weight, the mash temperature and / or the mash humidity. Furthermore, in addition to the monitoring of basic parameters such as temperature, humidity, and mash moisture, other key status information may be introduced, such as gas composition analysis, environmental pressure monitoring, and real-time flow detection, etc., to achieve a comprehensive perception of the production environment.

[0046] Preferably, using sensors such as infrared temperature sensors and digital temperature and humidity sensors can ensure accurate temperature and humidity detection of the mash and its surrounding environment; using capacitive or resistive moisture sensors can efficiently and accurately detect the moisture content of the mash.

[0047] Furthermore, if gas composition analysis is required, gas sensors such as the MQ series sensors can be used. These sensors can monitor the concentrations of gases such as carbon dioxide, carbon monoxide, and volatile organic compounds in the koji addition environment in real time, ensuring that no harmful gases accumulate during the koji addition process, thereby ensuring production safety and stability.

[0048] Preferably, based on the acquired real-time data (primarily the temperature and humidity of the mash), the end point of spreading (i.e., the start point of adding koji) can be determined by comparing it with the set initial temperature and humidity values, wherein the initial temperature and humidity values ​​can be set based on the combination of different types of mash and koji powder. Furthermore, when the mash is cooled by negative pressure spreading, when the end point of spreading is reached, the vacuum pump can be turned off to stop the suction, thereby stopping the spreading and starting the addition of koji.

[0049] For example, in one embodiment, the status information shown in the following table can be obtained through step S1:

[0050]

[0051] Preferably, in step S2, the amount of songs to be added can be accurately analyzed based on the acquired real-time data and the established mid-end decision layer DQN (Deep Q-Network) deep reinforcement learning algorithm to generate a corresponding song addition strategy.

[0052] Preferably, the present invention can use the Q function to represent the expected return of wine production by adding koji under the current production state, wherein the Q function can be used to evaluate the expected return of taking a certain action under a specific state, so as to help the intelligent agent judge the value of choosing a certain action under the current state, and thus make a better decision. Conventional Q functions are usually recorded as Q(s,a) or Q=(s,a), where s represents the current state of the environment and a represents the action taken under this state. The present invention applies the Q function to DQN, which is used to evaluate the value of actions under complex states. In this case, the Q function is a deep neural network, which can process high-dimensional state spaces. Preferably, the Q network of the present invention is as follows:

[0053]

[0054] Among them, s is the condition parameter under the current production environment; temperature is temperature; humidity is humidity; weight_of_dough is the weight of the mash; season is the season of the production environment; a represents the amount of koji added under the condition parameters of the current production environment; θ is the parameter of the Q network, which is used to evaluate the predicted wine production after adding koji powder with the calculated amount of koji under the current production condition parameters; R represents the value range of action a, that is, the action space is a continuous real number domain.

[0055] Preferably, the wine production conditions may be, for example, wine production rate and wine quality.

[0056] Preferably, on the basis of the Q network, the present invention also introduces a Q' network with the same structure as the Q network but different parameters, wherein the structure of the Q' network (i.e., the target network) can be regarded as the same deep neural network as the Q network (i.e., the main network), including the same number of layers and activation functions, but its weights and bias parameters remain fixed during the training process. This design enables the Q' network to provide a relatively stable target Q value for the current state-action pair, thereby helping to reduce instability during the training process. Since the parameters of the Q network are frequently updated, if the main network is used directly to calculate the target Q value, it may cause drastic fluctuations in the Q value estimate, thereby affecting the learning effect of the intelligent agent. By introducing the Q' network, the intelligent agent can rely on a fixed target network to calculate the target Q value, thereby avoiding noise and uncertainty caused by rapid parameter updates during the learning process. This approach not only improves the stability of training, but also helps to speed up the convergence speed, so that the intelligent agent can learn more effective strategies from the current environmental state and behavior. Preferably, the Q' network of the present invention is as follows:

[0057]

[0058] Where θ' is the parameter of the Q' network, which is used to stabilize and guide the learning process of the main network. It is regularly updated from the parameters θ of the main network to maintain parameter stability. τ is the update rate of the entire training process, usually between [0, 1], which controls the speed of updating the target network parameters. α is the learning rate, which controls the step size of each parameter update. is the gradient of the main network loss function L with respect to the parameter θ.

[0059] In the above formula, the second line indicates that the parameters θ' of the target network are updated by taking the weighted average of the parameters θ of the current main network and the parameters θ' of the previous target network; the third line indicates that the parameters of the main network are updated by gradient descent to minimize the loss function L(θ).

[0060] Preferably, the loss function L(θ) can be calculated using the following formula:

[0061] L(θ)=E[(yQ(s,a;θ)) 2 ]

[0062] Where y is the target value of the loss function, or the target Q value. The loss function L(θ) can be calculated by the variance between the target value of the loss function and the current Q network output.

[0063] Furthermore, the present invention may be provided with a reward function r for evaluating the wine production rate and wine quality corresponding to the current amount of koji added, and further evaluating its impact. Preferably, the reward function r can be expressed as: r(s, a; s'), wherein s' is the new state, in particular the new wine production state, which represents the predicted wine production rate and wine quality after the amount of koji added a is adopted. If the current amount of koji added can improve the wine production rate (i.e., produce more wine) and wine quality (i.e., the quality of the wine is better), the reward function will give positive feedback, otherwise it will give negative feedback. The goal of the intelligent agent is to maximize the cumulative reward, which means that throughout the production process, the intelligent agent not only pays attention to the current results, but also needs to consider possible future effects. When the intelligent agent chooses the amount of koji added a, it not only needs to consider the current state s, but also needs to evaluate the effect of its decision by predicting the new state s'. The design of the reward function enables the intelligent agent to optimize its koji addition strategy according to the changes in the wine production rate and wine quality, thereby achieving better production results. Therefore, the agent continuously tries different amounts of koji and adjusts its strategy based on the rewards it receives, learning how to choose the optimal amount to improve overall production results. This process forms a continuous learning and optimization cycle, ultimately helping the agent achieve optimal wine production rate and quality.

[0064] Preferably, when updating the Q network, the Q' network is used to calculate the target Q value, wherein the target Q value is usually calculated based on the maximum Q value of the current state and the next state, and its form is:

[0065] y=r+γmax a′ Q′(s′,a′;θ′)

[0066] Among them, y is the target Q value, which can be calculated by the Bellman equation. It provides a recursive way to define the state value function, which represents the wine production reward that can be obtained by the currently selected amount of added koji in a specific state; γ is the discount factor, which is used to achieve the trade-off between immediate rewards and future rewards, reflecting the value of future rewards relative to immediate rewards; a' is the optimal amount of added koji in the new (wine production) state s', that is, the optimal amount of added koji.

[0067] Preferably, the state-value function reflects the expected value of all possible future rewards after taking a certain action in a given state (i.e., performing the addition operation based on the currently selected addition amount). Through the Bellman equation, the agent can link the value of the current state with the value of the future state, so as to better evaluate the impact of different addition amounts on wine production benefits. In other words, the Bellman equation provides a systematic method that enables the agent to gradually update and optimize its state-value function based on the current addition amount and possible future results, and ultimately achieve the best wine production strategy. This recursive relationship helps the agent to learn and make decisions effectively in the complex decision-making process, thereby improving the overall efficiency of wine production. Preferably, the discount factor affects the agent's evaluation of the current addition strategy. A higher γ value means that this addition strategy will be considered more valuable in the future, and long-term execution is encouraged.

[0068] Preferably, the present invention may adopt the following greedy strategy to balance the behaviors of exploration and exploitation in the song adding strategy:

[0069]

[0070] Where ε is the exploration rate.

[0071] Preferably, the present invention can be configured with an experience replay buffer buffer = {(s, a, r, s')} to facilitate the extraction of samples from the experience replay buffer for training. The experience replay buffer is a crucial component of reinforcement learning, storing tuples of state, action, reward, and next state generated by the agent's interaction with the environment. This mechanism allows the agent to randomly draw from historical experience during training, rather than relying solely on recent experience, thereby effectively reducing correlations and improving learning efficiency.

[0072] Preferably, if Figure 2 As shown, in step S2, the training process of the model algorithm may include one or more of the following steps:

[0073] S2.1. Initialize Q network and Q' network;

[0074] S2.2. Create an experience replay buffer;

[0075] S2.3. Collect test data;

[0076] S2.4, select the action of adding volume;

[0077] S2.5. Storing the experience in the experience replay buffer;

[0078] S2.6, randomly sample experience from the experience replay buffer;

[0079] S2.7. Calculate the target Q value;

[0080] S2.8. Calculate the predicted value of the current Q network for the sampled data and update the Q network;

[0081] S2.9. Reduce the exploration rate and repeat the training;

[0082] S2.10. Training is complete, and the model is evaluated.

[0083] Preferably, in step S2.1, two Q-value networks can be defined first: one is the main network (i.e., the Q network), which is used for learning and updating; the other is the target network (i.e., the Q' network), which is used to stabilize the training process. Preferably, the Q network is usually initialized to zero or a small random value. Choosing a zero value can make the agent more conservative in the early stages of learning and avoid prematurely utilizing insufficient information. The initial weights of the Q' network can be the same as those of the Q network to ensure that their outputs are consistent at the beginning of training. The weights of the target network are updated regularly with the training cycle to ensure the stability of the learning process and reduce the oscillation of the Q value. Preferably, the Q network is usually constructed using a deep neural network, which receives state information at the input layer, the output layer corresponds to all possible additive actions, and the hidden layer is responsible for the abstract representation of features.

[0084] Preferably, in step S2.2, the experience replay buffer can be designed as a fixed-size FIFO (first-in, first-out) queue. When the experience replay buffer is full, the oldest experience is replaced by the new experience. Furthermore, the agent can store each experience (state, action, reward, and next state) in the experience replay buffer, so that a small batch of historical experience can be randomly sampled during subsequent training. By learning from multiple states and actions, the agent can better generalize to new, unseen states. In addition, the experience replay buffer allows the agent to use a richer sample when updating, thereby improving learning efficiency and stability.

[0085] Preferably, in step S2.3, the agent can collect the state information of the current production environment obtained in step S1. This information typically includes various factors that affect the amount of koji added, such as humidity, temperature, mash weight, and season. Furthermore, this state information of the current production environment can be combined into a state vector for subsequent input into the Q network for learning. Preferably, the collected data can be standardized or normalized before input into the neural network to ensure that different features have similar scales, thereby improving the effectiveness of training.

[0086] Preferably, in step S2.4, the agent may use an ε-greedy strategy to select a curvature action based on the current state, wherein the strategy can find a balance between exploration and exploitation so that the agent can learn the optimal strategy. Furthermore, when using the ε-greedy strategy, the agent may select the curvature action currently considered optimal in most cases, but may occasionally randomly select a curvature action for exploration, wherein the probability of randomly selecting a curvature action is the exploration rate ε; and the probability of selecting the optimal curvature action is 1-ε. Preferably, randomly selecting a curvature action helps to explore new strategies and states, thereby avoiding falling into local optimal solutions, and by selecting the curvature action with the largest Q value in the current Q network, the agent can leverage existing experience to maximize the expected reward.

[0087] Preferably, in step S2.5, after successfully selecting a wine-adding action, the agent can store the state information of the current production environment, the selected wine-adding action, the immediate reward and the new (wine-producing) state in the experience replay buffer to ensure that the agent can use these historical experiences for updating in the subsequent learning process.

[0088] Preferably, in step S2.6, the agent can periodically randomly extract a batch of experiences from the experience replay buffer for learning and updating, thereby effectively breaking the correlation between experiences and improving the generalization ability of the model. Through batch learning, the agent can utilize multiple experiences in a single update, thereby improving learning efficiency. The introduction of randomly sampled experiences solves the correlation problem in the learning process and avoids overfitting of the model due to reliance on the latest experience during training. By learning from a variety of different experiences, the agent can more comprehensively understand the value of state-action pairs, thereby promoting generalization. Batch learning not only improves computational efficiency, but also reduces noise during training, helping the model to stably converge to the optimal strategy.

[0089] Preferably, in step S2.7, after sampling the experience, the agent can calculate the target Q value according to the Bellman equation, wherein the target Q value is used to evaluate the expected reward that can be obtained by selecting a certain curvature action in a specific state and provide a benchmark for updating the Q network. The target Q value is the goal that the agent hopes to learn, and the output of the Q network should be as close to this value as possible. The target Q value helps the agent evaluate the quality of its behavioral strategy by combining immediate rewards and future expected rewards. The Bellman equation provides a recursive method for the agent to estimate the long-term rewards of curvature actions, ensuring that future impacts are taken into account during the learning process. By continuously updating the Q value, the agent can gradually optimize its strategy, thereby achieving better decision-making in complex environments.

[0090] Preferably, in step S2.8, after calculating the target Q-value, the agent can calculate the current Q-value network's prediction for the sampled data. This is used to assess the difference between the current policy and the target policy, thereby guiding subsequent updates. Furthermore, based on the current state and the curvature action, the agent can calculate the current Q-value using the Q-network. The difference between the current Q-value and the target Q-value is used to calculate the loss, thereby updating the parameters of the Q-network. The current Q-value reflects the agent's evaluation of a curvature action based on prior experience, while the target Q-value represents the goal the agent is striving for. By comparing the difference between the two, the agent can identify areas for improvement and gradually optimize its policy during training. This process ensures that the agent can flexibly adapt and make corresponding adjustments in a constantly updating environment. Furthermore, after calculating the current Q-value prediction and the target Q-value, the agent can update the parameters of the Q-network by minimizing the loss function to reduce the difference between the predicted and true values. Preferably, the agent can update the parameters of the Q-network using gradient descent or a variant thereof (such as the Adam optimizer). By minimizing the loss function, the agent can gradually approach the target Q-value, thereby optimizing its decision-making strategy. This process ensures the model's adaptability to environmental feedback, enabling the agent to continuously improve its decision-making ability in complex and dynamic environments. Effective parameter updates not only help improve the model's accuracy but also accelerate its convergence, allowing the agent to reach the optimal strategy as quickly as possible.

[0091] Preferably, in step S2.9, the agent can reduce the exploration rate ε to allow for greater utilization of existing knowledge in the later stages of learning. By gradually reducing exploration, the agent can more robustly execute the learned strategy. Preferably, the agent can set an initial and final exploration rate, as well as a decay factor, and adjust the exploration rate based on the decay factor after each training cycle, thereby enabling repeated training. Furthermore, a high exploration rate can be maintained in the early stages of training to ensure that the agent can explore a wide range of different strategies; as learning deepens, the exploration rate can be gradually reduced to increase the utilization of learned knowledge. A high exploration rate enables the agent to acquire rich data in the early stages of training, helping it understand the environment and optimize its strategy. In the later stages of training, reducing the exploration rate can reduce the model's randomness in known states, ensuring that the agent can stably execute the optimal strategy. This process is designed to balance exploration and exploitation, allowing the agent to fully utilize existing experience during learning while continuously seeking new areas for improvement.

[0092] Preferably, in step S2.10, the agent can evaluate the trained model algorithm on the test set to verify its reliability and stability under different environmental conditions. Preferably, a variety of evaluation indicators are designed during the evaluation, such as accuracy, stability, convergence speed, etc., to comprehensively evaluate the model performance, and these indicators can be calculated by comparing the differences between the model output and the actual output. Furthermore, cross-validation can be used to ensure the consistency of the model's performance on different data sets to avoid overfitting. Furthermore, the model algorithm is tested on a small scale in a real production environment to observe its optimization effect on the amount of added music, and the model algorithm is further adjusted based on the test results.

[0093] Ideally, once the model has been evaluated and meets the expected performance criteria, the agent's strategy can be deployed in the actual brewing process. The established model algorithm can be used to accurately calculate the optimal amount of koji to add, thereby generating a koji addition strategy.

[0094] For example, in one embodiment, state s is as follows: the production environment temperature is 15°C, the production environment humidity is 70%, the production environment season is spring, the mash weight is 500 kg, the mash temperature is 35°C, and the mash humidity is 55%. Based on this, the amount of koji added for action a is set to 50 kg, 55 kg, 60 kg, 65 kg, and 70 kg. After calculation in step S2, the Q-network output Q-values ​​for action a corresponding to these five different amounts of koji added are 10.5, 11.0, 12.5, 11.8, and 10.0, respectively. Therefore, the Q-value corresponding to the amount of koji added of 60 kg is the highest, at 12.5. This means that, in the current state, choosing 60 kg of koji adds the best wine yield and quality. In other words, the agent can choose 60 kg as the amount of koji added in order to obtain the highest reward. Furthermore, according to the greedy strategy, if the agent chooses to exploit, it will choose 60 kg; if the agent chooses to explore, it will try other amounts of koji added. After executing the action, the Q network is updated according to the actual reward obtained to continuously optimize the song addition strategy.

[0095] Preferably, in step S3, the koji addition operation is performed according to the koji addition strategy generated in step S2, wherein the change in the weight of the koji powder in the koji addition container 320 can be monitored by weighing to determine the end point of koji addition. Preferably, when the actual amount of koji added reaches the calculated optimal amount of koji added, that is, when the end point of koji addition is reached, the koji addition container 320 can be closed to stop the koji addition operation.

[0096] Example 2

[0097] This embodiment is a further improvement of embodiment 1, and repeated contents will not be repeated here.

[0098] like Figure 3As shown, the present invention also discloses an intelligent control song adding system, which includes a detection module 100, a control module 200 and an execution module 300, wherein the control module 200 is capable of executing the intelligent control song adding method as described in Example 1, and outputting a corresponding control signal to the execution module 300 to complete the song adding operation.

[0099] Preferably, the detection module 100 may include various sensors, such as temperature sensors and humidity sensors, to perform the front-end detection process described in step S1 of Example 1 and obtain corresponding data information. Furthermore, the data information obtained by the detection module 100 may be sent to the control module 200, so that the control module 200 can calculate the optimal amount of song addition based on the data information obtained by the detection module 100, thereby driving the execution module 300 to perform the corresponding song addition operation according to the song addition strategy included in the control signal.

[0100] Preferably, the detection module 100 may also include a weighing unit 110 for obtaining weight data, so that when the execution module 300 performs the corresponding song addition operation according to the song addition strategy, the control module 200 can determine the song addition end point based on the weight data obtained in real time by the weighing unit 110.

[0101] Preferably, if Figure 4 As shown, the execution module 300 may include a fan 310, a koji adding container 320, an air supply duct 330, and a sealed tank 340. The air supply duct 330 may be connected to the fan 310 at one end and to the sealed tank 340 at the other end. The koji adding container 320 can be connected to the air supply duct 330 in a controllable opening and closing manner, allowing the koji powder in the koji adding container 320 to enter the sealed tank 340 in a controlled manner through the air supply duct 330. Preferably, a weighing unit 110 may be connected to the koji adding container 320, and in particular, may be positioned below the koji adding container 320 to monitor the weight change of the koji adding container 320, thereby calculating the actual amount of koji added in real time. Furthermore, when the actual amount of koji added reaches the optimal amount of koji added calculated by the control module 200, the control module 200 may close the koji adding container 320 to stop the koji adding operation.

[0102] Preferably, the execution module 300 may be provided with a locking unit on the closed tank body 340, so that after adding mash into the closed tank body 340, the lid body is locked by the locking unit to ensure that the entire koji adding process is completely in a closed state, effectively solving the problem of koji powder dust overflowing during the conventional koji adding process.

[0103] Preferably, the execution module 300 may further include a flipping unit and / or a stirring unit, wherein the above two functional units may be set independently or simultaneously to achieve mixing of the koji powder and the material in the closed tank 340 .

[0104] In one embodiment, the control module 200 can be set to start the koji addition process when the temperature of the mash drops to 18-22°C after airing. The detection module 100 can send the acquired data information such as the production environment temperature and humidity to the control module 200, so that the control module 200 can use the established model algorithm to accurately analyze the amount of koji added, thereby outputting the optimal amount of koji added and generating a corresponding koji addition strategy. The execution module 300 can perform the corresponding koji addition operation according to the koji addition strategy and close the koji addition container 320 under the control of the control signal generated by the control module 200 when the koji addition end point is reached. Preferably, when the koji addition end point is about to be reached, the execution module 300, controlled by the control signal generated by the control module 200, can reduce the outlet of the koji addition container 320 to achieve micro-adjustment of the final koji addition process. The intelligent control koji addition system of the present invention can accurately control the amount of koji added to 0.1%.

[0105] It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention specification and its drawings are illustrative and do not constitute a limitation on the claims. The scope of protection of the present invention is defined by the claims and their equivalents. The present invention specification contains multiple inventive concepts, such as "preferably" or "according to a preferred embodiment", which means that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application based on each inventive concept. Throughout the text, the features guided by "preferably" are only an optional method and should not be understood as having to be set. Therefore, the applicant reserves the right to abandon or delete the relevant preferred features at any time.

Claims

1. An intelligent control method for adding songs, characterized in that: It includes: Front-end detection: Detect the status information related to the song addition process to obtain data information for analysis; Mid-end decision-making: Calculate the number of songs to be added based on the acquired data to generate a song addition strategy; Backend output: Execute corresponding song adding operations based on the generated song adding strategy. Among them, in the mid-end decision-making process, the DQN deep reinforcement learning algorithm is established and trained to evaluate the potential impact of different koji addition amounts on wine production rate and wine quality under the current production status obtained in the front-end detection process, so as to determine the expected return of taking a specific koji addition action, thereby determining the optimal koji addition amount under a specific production state and generating a dynamic koji addition strategy for execution by the back-end output process. In the mid-end decision-making process, the Q network is defined and trained into an approximate Q function to quantify the expected return of taking a specific koji addition action under a specific production state, and then the Q' network is introduced. A network is used to stabilize the learning process, wherein the state information of the current production state can be combined into a state vector to facilitate subsequent input into the Q network for learning, and the state information of the current production state includes state information related to the production environment and / or state information related to the mash, wherein the state information related to the production environment includes the production environment temperature, the production environment humidity and / or the production environment season; the state information related to the mash includes the mash weight, the mash temperature and / or the mash humidity, and the acquired state information can be input into the Q network after being standardized or normalized.

2. The intelligent control song adding method according to claim 1, characterized in that: An ε-greedy strategy is used to select a bending action based on the current production state, where the bending action is randomly selected with a probability based on the exploration rate ε or the optimal bending action is selected with a probability of 1-ε; and the experience corresponding to the selected bending action is stored in the experience playback buffer, where the experience includes the state information of the current production environment, the selected bending action, the immediate reward and the new state.

3. The intelligent control song adding method according to claim 2, characterized in that: A batch of experiences are randomly sampled from the experience replay buffer for learning and updating at regular intervals. After sampling the experiences, the target Q value is calculated according to the Bellman equation. The target Q value is used to evaluate the expected reward that can be obtained by selecting a certain action under a specific state and provides a benchmark for updating the Q network.

4. The intelligent control song adding method according to claim 3, characterized in that: Based on the current production state and the amount of curvature added, the current Q value is calculated by the Q network, and the loss is calculated by minimizing the loss function according to the difference between the current Q value and the target Q value, thereby updating the parameters of the Q network by the gradient descent method or its variants.

5. The intelligent control song adding method according to claim 4, characterized in that: By setting an initial exploration rate and a final exploration rate, as well as a decay factor, and adjusting the exploration rate according to the decay factor after each training cycle, the exploration is gradually reduced until the training is completed. The training set is used to evaluate the trained model algorithm to verify its reliability and stability under different environmental conditions.

6. The intelligent control song adding method according to claim 5, characterized in that: During the back-end output process, the corresponding koji adding operation is performed according to the generated koji adding strategy, and the change in the koji powder weight in the koji adding container (320) is monitored by weighing to determine the koji adding end point. When the koji adding end point is reached, the koji adding container (320) is closed to stop the koji adding operation.

7. An intelligent control song adding system, characterized in that: It includes: A detection module (100) is used to detect status information related to the song adding process to obtain data information for analysis; A control module (200) is used to calculate the amount of songs to be added based on the acquired data information to generate a song adding strategy; An execution module (300) is used to execute a corresponding song adding operation based on the generated song adding strategy, The control module (200) is capable of executing the intelligent control song adding method according to any one of claims 1 to 6, and outputting a corresponding control signal to the execution module (300) to complete the song adding operation.

8. The intelligent control song adding system according to claim 7, characterized in that: The execution module (300) includes a fan (310), a koji adding container (320), an air supply duct (330) and a closed tank body (340), wherein one end of the air supply duct (330) is connected to the fan (310), and the other end thereof is connected to the closed tank body (340), and the koji adding container (320) connected to the weighing unit (110) configured with the detection module (100) can be connected to the air supply duct (330) in a controllable opening and closing manner, so that the koji powder in the koji adding container (320) can enter the closed tank body (340) through the air supply duct (330) in a controlled manner.

Images