Mine large model reinforcement learning optimization method, device and medium based on adaptive advantage estimation

Abstract

The application provides a mine large model reinforcement learning optimization method based on adaptive advantage estimation, equipment and medium. The method comprises the following steps: obtaining a plurality of candidate response sequences for mine safety queries, and performing static weight assignment on each word in each candidate response sequence based on a mine safety field vocabulary; calculating the dynamic weight of each word based on the gradient information of the policy model when each word is generated; fusing the static weight and the dynamic weight to generate the fused weight of each word according to the current training stage; weighting the relative advantage estimation value in the group based on the fused weight to generate the adaptive advantage estimation value of each word, and updating the parameters of the policy model by using the adaptive advantage estimation value. The method can significantly improve the efficiency of the mine large model in the reinforcement learning training, reduce the iteration steps, and effectively improve the identification accuracy of the major accident risk and hidden risk clues.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology for large language models in mining, and in particular to a reinforcement learning optimization method, device and medium for large mining models based on adaptive advantage estimation. Background Technology

[0002] Currently, large language models for mining, as a key supporting technology for intelligent mining construction, are widely used in core scenarios such as safety risk identification, emergency response decision-making, and procedure consultation. Among related technologies, an optimization framework for large language models has been constructed through the collaborative operation of the Group Relative Policy Optimization (GRPO) algorithm and reinforcement learning post-training. Specifically, this framework covers the entire process from query input and multi-response sampling to within-group relative advantage estimation, including key aspects such as reward function design, normalization processing, and policy gradient update, aiming to improve the generation quality of models in specialized domains.

[0003] However, the GRPO method in related technologies directly adopts a sequence-level uniform reward distribution mechanism, which does not achieve token-level differential advantage estimation. This may result in safety-critical tokens (such as "gas outburst" and "emergency evacuation") receiving the same optimization signal as ordinary tokens, or in the early stages of training, the homogeneity of reward distribution may cause a "gradient starvation" phenomenon, thereby affecting the model's ability to learn the fine-grained importance of risk levels and semantic significance in mine safety knowledge, resulting in slow improvement in safety response accuracy and unstable convergence. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the first objective of this application is to propose a reinforcement learning optimization method for large mining models based on adaptive advantage estimation.

[0006] The second objective of this application is to propose an electronic device.

[0007] The third objective of this application is to provide a computer-readable storage medium.

[0008] To achieve the above objectives, the first aspect of this application is to propose a reinforcement learning optimization method for large-scale mining models based on adaptive advantage estimation, comprising the following steps:

[0009] Multiple candidate response sequences for a mine safety query are obtained, and static weights are assigned to each term in each candidate response sequence based on a pre-built mine safety domain thesaurus. The static weights reflect the risk level of the term in the mine safety domain. Based on the gradient information of the policy model when generating each word, the dynamic weight of each word is calculated. The dynamic weight is adaptively adjusted according to the training stage. In the early stage of training, the gradient norm of the word is emphasized, and in the later stage of training, the contribution of the word is emphasized. Based on the current training stage, the static weights and the dynamic weights are fused to generate fused weights for each word element. In the early stage of training, the fusion coefficient of the static weights is higher than that of the dynamic weights, and in the later stage of training, the fusion coefficient of the dynamic weights is higher than that of the static weights. The relative advantage estimates within a group are weighted based on the fusion weights to generate adaptive advantage estimates for each word, and the parameters of the strategy model are updated using the adaptive advantage estimates.

[0010] To achieve the above objectives, a second aspect of this application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the mine large model reinforcement learning optimization method based on adaptive advantage estimation as described in any one of the first aspects above.

[0011] To achieve the above objectives, the third aspect of this application also proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the reinforcement learning optimization method for large mining models based on adaptive advantage estimation as described in any of the first aspects above.

[0012] The technical solution provided by the embodiments of this application brings at least the following beneficial effects: This application improves the training efficiency of large models, significantly increases the coverage of early effective guidance, the number of iterations required to reach the target safety response accuracy, and the focus accuracy of later optimization, enhances the safety recognition performance of the trained large mining model, strengthens the fine-grained semantic learning of the large mining model, and significantly improves the accuracy of the model's speech understanding. Therefore, this application can significantly improve the training efficiency of reinforcement learning for large mining models, effectively solve the gradient starvation and credit allocation problems caused by the uniform distribution of token-level reward signals in the GRPO algorithm, enhance the model's fine-grained recognition ability of safety-critical tokens, and improve the accuracy of major accident risk identification and the recall rate of hidden risk clues.

[0013] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0014] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 The flowchart shows a reinforcement learning optimization method for a large mining model based on adaptive advantage estimation proposed in this application. Figure 2 This is a schematic diagram of the algorithm flow for a multi-adaptive advantage estimation mechanism proposed in an embodiment of this application; Figure 3 This is a schematic diagram illustrating a specific large-scale mining model training process proposed in an embodiment of this application. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0016] The following description, with reference to the accompanying drawings, illustrates a reinforcement learning optimization method for large mining models based on adaptive advantage estimation, as proposed in this application.

[0017] Example 1 Figure 1 This is a flowchart of a reinforcement learning optimization method for large-scale mining models based on adaptive advantage estimation, as proposed in an embodiment of this application. Figure 1 As shown, the method includes the following steps: Step S101: Obtain multiple candidate response sequences for the mine safety query, and assign static weights to each term in each candidate response sequence based on a pre-built mine safety domain thesaurus. The static weights reflect the risk level of the term in the mine safety domain.

[0018] Specifically, this application first obtains multiple candidate response sequences for a mine safety query, and then assigns static weights to each term in each candidate response sequence based on a pre-built mine safety domain lexicon. These static weights characterize the risk level of each term in the mine safety domain. The core of this approach lies in using prior domain knowledge to differentiate and label different terms in the generated sequences, thus laying the foundation for subsequent fine-grained optimization.

[0019] Specifically, by constructing a professional thesaurus covering core terms, risk descriptions, and key concepts in the field of mine safety, and establishing a hierarchical mapping relationship based on the risk level corresponding to each term, high-risk terms receive higher static weights, while low-risk or neutral terms receive lower weights. The assignment of these static weights can be achieved through various methods such as exact matching, prefix matching, or semantic similarity calculation to ensure comprehensive coverage and reasonable differentiation of various terms in the sequence.

[0020] As one possible implementation, the mine safety terminology can be constructed based on the fusion of multi-source data such as regulations and standards, historical accident cases, and domain expert knowledge. It adopts a non-linear mapping relationship between four-level risk levels and static weights. For example, it assigns higher weights to terms related to major emergency risks and lower weights to terms related to basic safety concepts, thereby achieving a preliminary quantification of the safety importance of different terms in the sequence.

[0021] Therefore, by introducing a static weight assignment mechanism based on domain prior knowledge, this step effectively overcomes the problem of insufficient fine-grained semantic discrimination caused by the sequential-level uniform distribution of reward signals in traditional methods. This allows the model to obtain differentiated optimization guidance for high-risk words in the early stages of training, thereby significantly improving the sensitivity and learning efficiency in identifying key concepts for mine safety. Step S102: Based on the gradient information of the policy model when generating each word, calculate the dynamic weight of each word. The dynamic weight is adaptively adjusted according to the training stage. In the early stage of training, the gradient norm of the word is emphasized, and in the later stage of training, the contribution of the word is emphasized.

[0022] Specifically, this application calculates the dynamic weight of each word based on the gradient information generated by the strategy model. This dynamic weight is not fixed, but is adaptively adjusted according to the training stage to give different words different optimization attention at different training periods.

[0023] Specifically, the calculation of these dynamic weights relies on the analysis of the gradient norm generated by the policy model when generating each lexical. The magnitude of the gradient norm reflects the potential impact of that lexical on parameter updates in the current model state. In the early stages of training, before the model parameters have fully converged, the calculation of dynamic weights focuses on lexicals with larger gradient norms—those that can generate stronger gradient signals—thereby guiding the model to quickly explore the parameter space and establish a preliminary understanding of the basic concepts of the domain.

[0024] Furthermore, as the training process progresses to the later stages, the model has acquired certain basic capabilities, and the dynamic weight calculation strategy is switched accordingly, focusing on words with higher contribution, that is, those words that have a greater impact on the quality of model output or security relevance, thereby achieving fine-tuning of the strategy space.

[0025] As one possible implementation, in the early stages of training, dynamic weights can be calculated using a nonlinear function based on the statistical distribution of the gradient norm, such as an expression incorporating gradient norm normalization and a nonlinear activation function. The enhancement coefficient can be set within a specific interval to amplify the influence of high-gradient terms. In the later stages of training, a weight calculation method based on contribution estimation can be used, for example, by narrowing the enhancement coefficient interval to make the model focus on key terms and weaken external guidance.

[0026] In addition, an adaptive threshold mechanism based on gradient distribution statistics can be introduced. By maintaining the historical record of the gradient norm within the sliding window, its quantile distribution can be calculated in real time, and the threshold for dynamic weight calculation can be set to a quantile that decreases with the training process. This allows for dynamic adjustment of the proportion of lexical units assigned high dynamic weights, ensuring the stability and effectiveness of the training process.

[0027] Therefore, this step, by introducing an adaptive dynamic weight calculation mechanism during the training phase, achieves dynamic control over the optimization process of the policy model, enabling the model to focus on different types of lexical units at different training periods. This significantly improves training efficiency and optimization accuracy, avoiding the problems of weak gradient signals or wasted optimization resources under the traditional uniform distribution mechanism.

[0028] Step S103: Based on the current training stage, the static weights and dynamic weights are fused to generate fused weights for each word. In the early stage of training, the fusion coefficient of the static weights is higher than that of the dynamic weights, and in the later stage of training, the fusion coefficient of the dynamic weights is higher than that of the static weights.

[0029] Specifically, the core of the fusion process, which combines static and dynamic weights, lies in dynamically adjusting the contribution ratio of the two types of weights based on the training progress.

[0030] Specifically, in the early stages of training, since the model has not yet fully established its understanding of key concepts in the field of mine safety, the fusion coefficient of the static weights is set to be higher than that of the dynamic weights. This ensures that the safety vocabulary information based on domain prior knowledge can dominate the allocation of optimization signals, guide the model to quickly focus on high-risk terms, and establish a preliminary safety knowledge base.

[0031] As training progresses, the model's ability to recognize safety concepts gradually improves, the guiding role of static weights weakens accordingly, and the fusion coefficient of dynamic weights gradually increases. This allows real-time learning signals based on gradient feedback to play a larger role in weight fusion, thus enabling the model to adaptively adjust its optimization focus according to its own learning state.

[0032] In the later stages of training, the fusion coefficient of the dynamic weights is set higher than that of the static weights. At this point, the model has a relatively mature safety knowledge system, and optimization resources are more inclined to those terms that the model has not yet fully mastered or that have high learning value, in order to achieve fine-tuning of the policy space. This fusion process is implemented through a fusion coefficient function that changes with the number of training iterations. This function assigns a high constant coefficient to the static weights in the early stages of training, linearly decreases to a lower value in the middle stages, and maintains a low constant coefficient in the later stages, thus smoothly completing the transition from prior knowledge guidance to dynamic learning feedback dominance.

[0033] As one possible implementation, the fusion coefficient can be set to 0.7 in the early stage of training, linearly decay from 0.7 to 0.2 in the middle stage of training, and remain at 0.2 in the later stage of training. Accordingly, the static weights and dynamic weights are weighted and summed according to this coefficient, and the final fusion weights are obtained after processing by the alignment function.

[0034] Therefore, this step, by introducing an adaptive weight fusion strategy during the training phase, achieves the synergistic optimization of prior knowledge and dynamic learning signals, effectively solving the limitations of a single weight mechanism at different training stages. This accelerates the establishment of safety knowledge in the early stages of training and improves the precision of optimization in the later stages of training, thereby improving the overall efficiency of model training and final performance.

[0035] Step S104: The relative advantage estimates within the group are weighted based on the fusion weights to generate adaptive advantage estimates for each word, and the parameters of the strategy model are updated using the adaptive advantage estimates.

[0036] Specifically, after obtaining the fusion weight of each word, this application further uses the fusion weight to weight the relative advantage estimate within the group in order to generate an adaptive advantage estimate for each word.

[0037] Specifically, the intra-group relative advantage estimate is a sequence-level scalar value obtained by normalizing the reward distribution of multiple candidate response sequences corresponding to the same query. It reflects the relative superiority or inferiority of the candidate response within the entire group. However, this sequence-level advantage estimate cannot distinguish the differentiated contributions of different terms within the same sequence to the final reward. To address this, this application applies the aforementioned calculated fusion weights to the sequence-level advantage estimate, decomposing and distributing the sequence-level advantage signal to each term through weighted operations, thereby forming a term-level adaptive advantage estimate. This weighting process allows terms with higher fusion weights (i.e., those terms with higher risk levels in the field of mine safety or higher learning value in the current training stage) to receive amplified advantage signals, while terms with lower fusion weights receive relatively suppressed advantage signals, thus achieving fine-grained control over the policy gradient update direction.

[0038] Subsequently, these word-level adaptive advantage estimates are used to replace the traditional uniform advantage values ​​and participate in the calculation of policy gradients and the updating of model parameters, so that the policy model can apply differentiated learning pressure to different words during the optimization process.

[0039] As one possible implementation, the weighting process can be achieved by multiplying the sequence-level advantage estimate by the ratio of the fusion weight of each word to the average fusion weight of all words in the sequence, thereby achieving differentiated allocation of advantage signals among words while keeping the sequence-level advantage expectation unchanged.

[0040] Therefore, through the above-mentioned word-level advantage estimation generation and parameter update mechanism based on fusion weights, this application can break through the limitations of traditional sequence-level uniform distribution of advantage signals, realize the key leap from coarse-grained optimization to fine-grained optimization, significantly improve the sensitivity and learning efficiency of the policy model to key semantic information in the field of mine safety, and effectively alleviate the gradient hunger and credit allocation problems caused by the homogenization of advantage signals.

[0041] The following describes in detail the specific implementation process of each step in the mine large-scale model reinforcement learning optimization method based on adaptive advantage estimation of this application, using several specific embodiments. The principle of the gas emission prediction method of this application is as follows: Figure 2 As shown.

[0042] Example 2 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S101, "assigning static weights to each term in each candidate response sequence based on a pre-built vocabulary for the field of mine safety".

[0043] In this embodiment, the system first uses the Aho-Corasick automaton to precisely match each term in the candidate response sequence, querying whether the term has a direct entry in the mine safety domain thesaurus. This mine safety domain thesaurus adopts a hierarchical organizational structure, with each term labeled with its respective safety subdomain, risk level attribute, and typical contextual pattern. The risk levels include Level 1 Major Emergency Risk, Level 2 Significant Risk, Level 3 General Risk, and Level 4 Basic Safety Concept, with corresponding static weights of 1.2, 1.12, 1.08, and 1.0, respectively.

[0044] If a term has a direct entry in the thesaurus, the risk level and static weight corresponding to that term are directly obtained. For terms that do not have a direct entry, the system further checks whether the term is a prefix of a compound term in the mine safety thesaurus. If so, it assigns a value according to the prefix importance decay rule. For example, if the term is a prefix of a compound term, it is assigned a decay value lower than the original term weight based on its position and importance in the term.

[0045] For lexical units that do not match a direct entry or are not prefixes, the system uses a pre-trained language model encoder to calculate the semantic similarity between the lexical unit and each entry in the mining safety lexicon, and assigns a downgraded extended weight to lexical units with high similarity. For example, if the semantic similarity between a lexical unit and a certain safety term exceeds a preset threshold, then the lexical unit is assigned a proportionally reduced weight.

[0046] Finally, the system combines the local context of each term to determine whether it falls within a safety risk description context, and performs context-aware weight fine-tuning. For example, if a term appears in a context describing risk scenarios such as gas concentration or roof pressure, its weight is appropriately increased. Through these steps, the system ultimately outputs a static weight vector of the same length as the candidate response sequence. ,in, .

[0047] Therefore, this embodiment achieves accurate identification and differentiated weight allocation of key risk terms in the field of mine safety through a multi-level term matching and weight assignment mechanism, thereby providing reliable prior knowledge guidance for subsequent advantage estimation and significantly improving the model's sensitivity to key safety information.

[0048] Example 3 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S102, "calculating the dynamic weight of each word based on the gradient information of the strategy model when generating each word."

[0049] In this embodiment, the calculation of dynamic weights in step S102 employs differentiated strategies depending on the training stage. Specifically, in the early stages of training, when the number of iterations k is less than 20% to 30% of the total steps, a high-sensitivity-first dynamic weight strategy is adopted. The input source for this strategy is the gradient information of the strategy model when generating the current word, specifically the gradient norm of the log probability with respect to the model parameters.

[0050] The processing steps include: First, calculating the mean μ of the gradient norm of all lexical units within the batch. g and standard deviation σ gThen, the gradient norm of the current lexical unit is standardized by dividing its difference from the mean by the standard deviation. Next, the standardized value is mapped to the (0,1) interval using the sigmoid function. Finally, it is multiplied by the early reinforcement coefficient η and incremented by 1 to obtain the dynamic weight of the lexical unit. The early reinforcement coefficient η ranges from [0.1, 0.2]. The output of this process is the dynamic weight value of lexical unit t, calculated using the following formula:

[0051] Understandably, this design assigns higher dynamic weights to terms with larger gradient norms, encouraging the model to actively explore high gradient directions in the early stages of training, quickly escaping low-quality regions of the initial parameter space, and accelerating the initial establishment of a basic understanding of domain safety concepts. In the later stages of training, when the iteration count k is greater than or equal to this threshold, a dynamic weight strategy prioritizing high contribution is switched. At this point, the early reinforcement coefficient η is reduced to the [0.05, 0.1] range, tilting optimization resources towards terms that are uncertain but potentially important to the model, achieving a refined exploration of the policy space. Through this gradient-statistic-based adaptive threshold mechanism, the dynamic weight module can automatically adjust its focus according to the training progress, ensuring that the optimization signal always focuses on the terms with the highest learning value at the current stage.

[0052] Therefore, this specific implementation method, by introducing an adaptive dynamic weight strategy based on gradient norm statistics, achieves rapid focusing on high-gradient words in the early stage of training, effectively alleviates the gradient hunger problem caused by the homogenization of reward signals in the early stage of training of the standard GRPO algorithm, and significantly improves the effective guidance coverage and model convergence speed in the early stage of training.

[0053] In this embodiment, the dynamic weight calculation in step S102 during the later stages of training adopts a high contribution priority strategy. Specifically, when the number of training iterations k reaches or exceeds the preset number of warm-up steps K, the weight calculation is performed with a high contribution priority strategy. warmup When this happens, the system automatically switches to this strategy, where K warmup Typical setting is the total number of training steps K total The proportion is between 20% and 30%. In this stage, the input for calculating the dynamic weights is the gradient norm of the policy model when generating each word, the processing action estimates its contribution based on the statistical distribution of the gradient norm, and the output is the dynamic weight value of each word.

[0054] Specifically, dynamic weights The calculation formula is: ,in, To enhance the coefficient, its value range is reduced to [0.05, 0.1], which is significantly lower than the value range of [0.1, 0.2] in the early stage of training, in order to weaken external guidance and make the model focus on the key words. and These are the mean and standard deviation of the gradient norms of all lexical units in the current batch, respectively, used to standardize the gradient norms.

[0055] This processing action assigns higher dynamic weights to terms with gradient norms significantly higher than the batch average, thereby allocating optimization resources to terms that are "uncertain but potentially important," enabling fine-grained exploration of the policy space. The output of this sub-step is a dynamic weight vector of the same length as the generated sequence. Where G is the number of candidate responses, and T max The maximum sequence length is set. This dynamic weight vector is then input into the fusion calculation module in step S103, where it undergoes stage-adaptive fusion processing with the static weights. This is achieved by using the enhancement coefficients... By limiting the range to a small range of [0.05, 0.1], this embodiment effectively avoids the risk of over-amplifying gradient noise in the later stages of training, while ensuring that high-contribution terms can obtain sufficient optimization signals, thereby achieving stable and efficient parameter updates in the fine-tuning stage.

[0056] Therefore, this specific implementation reduces the reinforcement coefficient in the later stage of training to the range of [0.05, 0.1], thus achieving a smooth transition from early strong guidance to later autonomous focusing, effectively suppressing the amplification effect of gradient noise, and improving the training stability and convergence accuracy of the model in the fine optimization stage.

[0057] In this embodiment, the calculation of dynamic weights in step S102 further introduces an adaptive threshold mechanism based on gradient distribution statistics to ensure the stability of the training process and improve the allocation efficiency of optimization resources.

[0058] Specifically, the system maintains a sliding window to record and update the statistical information of the gradient norm of each word within a batch in historical training iterations. This sliding window stores the gradient norm data from the most recent few steps in real time and calculates the quantile distribution of the gradient norm at the current moment based on this data. The threshold used for dynamic weight calculation, i.e., the critical value used to determine whether a word belongs to a high gradient norm or a high contribution, is set as a quantile that decreases as the training progresses.

[0059] In the early stages of training, when the number of iterations accounts for 0% to 20% of the total steps, the system uses the 75th percentile as the threshold. This means that only words whose gradient norm falls within the top 25% of all words are assigned higher dynamic weights, thus focusing on the strongest signals and quickly establishing a preliminary understanding of basic domain safety concepts. As training progresses to the middle stage, when the number of iterations accounts for 20% to 60%, the threshold is reduced to the 60th percentile. At this point, the coverage expands to words whose gradient norm falls within the top 40%, achieving a balance between exploration and utilization, and appropriately expanding the coverage of optimized signals. In the later stages of training, when the number of iterations accounts for 60% to 100%, the threshold is further reduced to the 50th percentile. At this point, the coverage expands to words whose gradient norm falls within the top 50%, achieving broad coverage and fine-tuning of the policy space. This adaptive threshold mechanism effectively avoids gradient explosion or vanishing problems caused by drastic changes in gradient norm distribution by dynamically adjusting the proportion of activated words, ensuring the stability of the training process. At the same time, this mechanism enables the optimization resources to automatically focus on the most valuable lexical units according to different training stages, achieving a smooth transition from early strong signal-driven to later extensive fine-tuning.

[0060] Therefore, this specific implementation method introduces an adaptive threshold mechanism based on gradient distribution statistics, which enables the dynamic weight calculation threshold to decrease adaptively with the training process. This effectively avoids the training instability caused by gradient explosion or vanishing, and significantly improves the efficiency of optimization resource allocation, so that the model can obtain stable and efficient learning signals at each stage of training.

[0061] Example 4 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S103, "merging the static weights and the dynamic weights to generate the fusion weights of each word element".

[0062] In this embodiment, the fusion process of static and dynamic weights in step S103 is specifically implemented through a fusion coefficient λ(k) that adaptively changes with the training process. This fusion coefficient λ(k) is defined as a function of the training iteration number k, and its calculation formula is: when k is less than 0.2 times K... total When k is 0.2 times K, λ(k) takes the value of 0.7; when k is 0.2 times K total Up to 0.6 times K total Between these values, λ(k) linearly decreases from 0.7 to 0.2. When k is greater than or equal to 0.6 times K... total When λ(k) is fixed at 0.2, where K totalThis represents the total number of training steps. The design of this function ensures that static weights dominate in the early stages of training (prior guidance period), with a fusion coefficient as high as 0.7. This guarantees that prior knowledge built based on a vocabulary in the field of mine safety can effectively guide the model to quickly establish a preliminary understanding of basic safety concepts.

[0063] As training progresses into the mid-stage (hybrid adaptation phase), the fusion coefficient decreases linearly from 0.7 to 0.2, with static weights gradually giving way to dynamic weights. This achieves a smooth transition from prior knowledge-guided to gradient-feedback-based adaptive optimization. In the late training stage (fine-tuning phase), the fusion coefficient stabilizes at 0.2, and dynamic weights become dominant, allowing optimization resources to focus on potentially important but currently uncertain word terms for fine-tuning.

[0064] After determining the fusion coefficient λ(k), the fusion weight of each word is further calculated based on this coefficient. Specifically, for the t-th word in the i-th candidate response sequence, the formula for calculating its fusion weight is as follows:

[0065] in, This represents the static weight of the term. This represents the dynamic weight of the term. The (·) symbol represents the alignment function. This alignment function aims to appropriately amplify the effective range of the dynamic weight of a term when it has a high static weight (i.e., it belongs to a high-risk, safe term). This ensures that these safe terminologies receive sufficient optimization attention in the later stages dominated by dynamic weights, creating a "double reinforcement" effect. Through the above calculations, a fusion weight vector of the same length as the generated sequence is finally output. Each element in this vector represents the fusion weight of the corresponding term, used for subsequent weighting of the relative advantage estimates within the group.

[0066] This specific implementation introduces a stage-adaptive fusion coefficient function and alignment function to achieve synergistic optimization of static prior knowledge and dynamic learning feedback. Its beneficial effects are: it rapidly injects domain safety knowledge in the early stages of training, avoiding training stagnation or erroneous cognition in the initial stage due to a lack of effective guidance; and in the later stages of training, it can dynamically allocate optimization resources to the most valuable lexical units according to the model's own learning state, thereby significantly improving training efficiency and enhancing the model's sensitivity and accuracy in recognizing and responding to safety keywords.

[0067] In this embodiment, the fusion process of static and dynamic weights in step S103 does not rely solely on a preset fixed iteration step threshold for stage switching. Instead, it further introduces an automatic triggering mechanism based on validation set performance monitoring. Specifically, the system continuously maintains a validation set during training, which includes several mine safety-related queries and their standard responses. At each preset evaluation interval (e.g., after a certain number of training steps), the system generates responses to the queries in the validation set using the current strategy model and calculates the safety response accuracy. This accuracy measures the correctness and risk identification capability of the model-generated content in terms of mine safety knowledge. The system records and tracks the trend of this safety response accuracy as the number of training steps increases and calculates its improvement rate. When the improvement rate is detected to be lower than a preset threshold, the system determines that the current training stage can no longer bring significant effective performance gains to the model. At this time, it will automatically trigger stage switching, allowing the training process to enter the next stage ahead of schedule.

[0068] For example, if the rate of improvement in safety response accuracy remains below a preset threshold during the prior guidance phase, the system will prematurely end this phase and enter the hybrid adaptation phase. Similarly, if the rate of improvement falls below the threshold during the hybrid adaptation phase, the system will prematurely enter the fine-tuning phase. This automatic triggering mechanism ensures the effective use of training resources, avoids wasting computational resources when model performance improvement has plateaued, and allows the adjustment of fusion coefficients to dynamically adapt to the actual learning progress of the model, thereby accelerating convergence and improving the final model performance overall.

[0069] Therefore, this embodiment introduces an automatic stage switching mechanism based on validation set performance monitoring, which enables adaptive adjustment of the training process, avoids the waste of training resources or stage switching delay caused by fixed step switching, thereby further improving training efficiency and enabling the model to converge to a better performance state more quickly.

[0070] Example 5 Based on the above embodiments, this embodiment provides a detailed description of the specific implementation of step S104, "weighting the relative advantage estimates within the group based on the fusion weights to generate adaptive advantage estimates for each word, and updating the parameters of the strategy model using the adaptive advantage estimates."

[0071] In this embodiment, the process of generating the adaptive advantage estimate in step S104 specifically involves weighting the relative advantage estimate within a group based on the fusion weights. First, the system obtains the fusion weights calculated for each word in each candidate response sequence in step S103. ,in Indicates the first A number of candidate response sequences Indicates the first in the sequence Each word. Simultaneously, the system acquires the estimated intra-group relative advantage value calculated using the standard GRPO algorithm. This value is based on the first The reward value of a candidate response sequence is a scalar obtained by normalizing the reward value of all responses in the group relative to the mean and standard deviation of the reward values.

[0072] Subsequently, the system addresses each candidate response sequence. Calculate the mean of the fusion weights of all tokens within the sequence. This mean reflects the overall weight level of the sequence. Based on this, the system calculates the fusion weight for each word. Divide by the weighted mean of its sequence This yields a normalized weighting factor.

[0073] Finally, the system compares the proportionality factor with the estimated relative advantage within the group. Multiply to generate an adaptive advantage estimate for the term. Its calculation formula is This calculation process ensures that for each candidate response sequence, the expected value of the adaptive advantage estimate for all its terms is equal to the original within-group relative advantage estimate, i.e. This allows for token-level differentiated advantage allocation while maintaining the overall optimization direction. Finally, the system utilizes the calculated adaptive advantage estimates for each lexical unit. It replaces the uniform advantage value in the standard GRPO and is used for calculating policy gradients and updating model parameters.

[0074] Therefore, this embodiment introduces a normalization scaling factor based on fusion weights to achieve token-level fine-grained weighting of the relative advantage estimates within a group. This allows safety-related keywords to receive optimized signal strength commensurate with their importance, while the optimized signals of non-keyword keywords are correspondingly weakened. This mechanism effectively solves the "gradient starvation" and credit allocation problems caused by the uniform distribution of reward signals in the standard GRPO algorithm, significantly improving the model's sensitivity and learning efficiency to high-risk terms in the mining safety field, thereby accelerating training convergence and enhancing the model's safety response accuracy.

[0075] Example 6 This embodiment provides a detailed description of a complete implementation of the aforementioned large-scale mine reinforcement learning optimization method, particularly focusing on the specific implementation process of the large-scale mine reinforcement learning optimization method based on adaptive advantage estimation. This method aims to overcome the problems of insufficient fine-grained optimization caused by sequence-level uniform reward allocation, lack of effective guidance in the early training stages, and low sensitivity in identifying safety-critical tokens in existing population relative policy optimization algorithms. By introducing a multi-adaptive advantage estimation mechanism, it achieves a crucial leap from sequence-level coarse-grained optimization to token-level fine-grained optimization.

[0076] It should be noted that this embodiment is based on the Multi-Adaptive Advantage Estimation Mechanism (MAAEM) proposed in this application. Its design goal is to overcome the limitations of the GRPO algorithm's token-level uniform optimization while retaining its computational efficiency advantages, thereby achieving fine-grained, differentiated advantage estimation for the mining safety field. This mechanism consists of three closely cooperating sub-modules: a static weight module provides prior guidance based on domain knowledge, a dynamic weight module implements adaptive adjustment based on learning feedback, and a fusion calculation module integrates dual signals to generate the final advantage estimate. The algorithm flow of this mechanism is as follows: Figure 2 As shown, the static weight module is used for high-risk token identification driven by a secure vocabulary, the dynamic weight module is used for gradient-based progressive adjustment, and the fusion calculation module is used for estimating the synergistic advantage of static and dynamic weights. The working process of this mechanism will be explained in detail below with specific examples.

[0077] First, the system receives two types of input data. The first type of input is the user query q, which covers various typical scenarios in the field of mine safety, including but not limited to risk identification, emergency response, procedure consultation, and accident analysis. The second type of input is a group of multiple candidate response sequences generated by the strategy model for this query, represented as follows: Where G is the number of responses within a group, typically 8 or 16. During the input processing phase, the system preprocesses the user query q, including intent recognition, risk level estimation, and keyword extraction, to provide contextual information for subsequent static weight calculation. For each candidate response sequence... It contains T tokens, and the sequence lengths may vary. The system will pad or truncate them to a uniform maximum length. .

[0078] In the static weight calculation branch, the system assigns static weights to each term in each candidate response sequence based on a pre-constructed thesaurus of mine safety. This thesaurus of mine safety is constructed using a three-source fusion strategy of "regulations and standards + accident cases + expert knowledge." Specific data sources include: 3,847 core terms extracted in a structured manner from authoritative regulations and standards such as the "Coal Mine Safety Regulations"; 2,156 characteristic terms extracted from typical accident investigation reports of the National Mine Safety Administration from 2010 to 2024 using natural language processing technology to describe accident causes, symptoms, and consequences; and 987 key prompts collected from frontline safety management and technical personnel through structured interviews and the Delphi method.

[0079] Multi-source data were fused using term alignment and disambiguation algorithms to ensure that different expressions of the same concept were normalized while retaining meaningful semantic variations. The final vocabulary contained 6,230 entries, organized hierarchically. Each entry was labeled with its respective safety subdomain (such as ventilation, gas, roof, water hazard, fire, electromechanical, transportation, etc.), risk level attribute, and typical contextual pattern. The risk levels are divided into four levels: Level 1 is a major emergency risk, defined as an extreme danger that could lead to mass casualties, with typical examples including "gas explosion," "coal and gas outburst," and "water inrush accident," and its static weight is assigned a value of 1.2; Level 2 is a significant risk, defined as a risk that could cause casualties or significant property damage, with typical examples including "roof pressure," "equipment operating with defects," and "ventilation system failure," and its static weight is assigned a value of 1.12; Level 3 is a general risk, defined as a hidden danger that requires attention but has a low urgency, with typical examples including "decreased support quality" and "high dust concentration," and its static weight is assigned a value of 1.08; Level 4 is a basic safety concept, defined as basic content such as safety management systems and operating procedures, with typical examples including "certified work," "pre-shift meeting," and "safety inspection," and its static weight is assigned a value of 1.0. The weight mapping adopts a non-linear design to ensure significant differentiation of high-risk levels. For risk expressions with multiple words, a risk modifier mechanism is introduced: when a negative word is combined with a risk word, the level of risk is increased by one level; degree adverbs adjust the basic weight proportionally; and time-urgent words receive an additional 10% bonus in emergency response scenarios.

[0080] In the specific static weight assignment process, the system executes the following steps: First, precise matching: The Aho-Corasick automaton is used to precisely match each term in the candidate response sequence to check if the term has a direct entry in the mine safety lexicon. If it does, the risk level and static weight corresponding to that term are directly obtained. Second, prefix matching: For terms that do not match a direct entry, it is checked whether the term is a prefix of a compound term in the mine safety lexicon. If so, a weight is assigned according to the prefix importance decay rule. Third, semantic expansion: For terms that do not match a direct entry and are not prefixes, a pre-trained language model encoder is used to calculate the semantic similarity between the term and each entry in the mine safety lexicon. Terms with high similarity are assigned a downgraded expanded weight. Fourth, context modulation: Combining the local context of each term's location, it is determined whether the term is within a safety risk description context, and context-aware weight fine-tuning is performed. Finally, the system outputs a static weight vector of the same length as the candidate response sequence. Each static weight The value range is [1.0, 1.2].

[0081] In the dynamic weight calculation branch, the system calculates the dynamic weight of each word based on the gradient information of the policy model when generating each word. This dynamic weight is adaptively adjusted according to the training phase. In the early stages of training, when the number of iterations k is less than 20% to 30% of the total steps, a high-sensitivity-first dynamic weight strategy is adopted. Specifically, the formula for calculating the dynamic weight is:

[0082] in, This is the early strengthening coefficient, with a value range of [0.1, 0.2]. and These represent the mean and standard deviation of the gradient norm within the batch, respectively. This design assigns higher dynamic weights to terms with larger gradient norms, encouraging the model to actively explore high gradient directions early in training and quickly escape low-quality regions of the initial parameter space. Later in training, when the number of iterations k is greater than or equal to 20% to 30% of the total steps, a dynamic weight strategy prioritizing high contribution is switched, where the enhancement coefficient... The value range is narrowed to [0.05, 0.1]. This design allows optimization resources to be tilted toward lexical terms that are "uncertain but potentially important," enabling fine-grained exploration of the policy space.

[0083] To avoid training instability caused by gradient explosion or vanishing, the system also incorporates an adaptive threshold mechanism based on gradient distribution statistics. The system maintains historical records of gradient norms within a sliding window, calculates their quantile distribution in real time, and sets the threshold for dynamic weight calculation to a quantile that decreases with the training progress. Specifically, in the early training phase (0% to 20%), a 75th quantile is used, covering the top 25% of words, with the intention of focusing on the strongest signals and quickly establishing a foundation. In the middle training phase (20% to 60%), a 60th quantile is used, covering the top 40% of words, with the intention of moderate expansion and balancing exploration and utilization. In the later training phase (60% to 100%), a 50th quantile is used, covering the top 50% of words, with the intention of broad coverage and fine-tuning.

[0084] During the fusion calculation phase, the system fuses static and dynamic weights based on the current training stage to generate fusion weights for each word. Fusion coefficients The function for training iteration k is calculated as follows:

[0085] in, Total training steps. Fusion weights. The calculation formula is:

[0086] in, The alignment function is used to appropriately amplify the effective range of dynamic weights for terms with high static weights. This fusion mechanism implements a multi-stage adaptive switching strategy: during the prior guidance period (iteration range)... ), fusion coefficient With a value of 0.7, static weights dominate, quickly establishing the foundation for a safety concept; while in the hybrid adaptation period (i.e., the iteration range is...), ... ), fusion coefficient The weight decreases linearly from 0.7 to 0.2, with static and dynamic weights balanced and integrated to explore the optimization space; during the fine-tuning period (iteration range)... ), fusion coefficient With a value of 0.2, dynamic weighting dominates, focusing on fine-tuning of high-contribution terms. Furthermore, stage switching can be automatically triggered by validation set performance monitoring; when the rate of improvement in safety response accuracy falls below a preset threshold, the system advances to the next training stage.

[0087] In the final dominance estimation stage, the system weights the relative dominance estimates within a group based on fusion weights to generate an adaptive dominance estimate for each word. Specifically, the adaptive dominance estimate for each word... The calculation formula is:

[0088] in, The estimated relative advantage within the group is calculated as follows:

[0089] in, The fusion weight for word elements, This is the mean of the fusion weights of all tokens in the candidate response sequence.

[0090] Furthermore, this weighted advantage estimate is directly used to calculate the policy gradient, replacing the uniform advantage value of the standard GRPO. This output possesses the following property: expectation preservation, i.e. Variance adaptability, meaning the variance of key tokens is significantly higher than that of non-key tokens; dynamic adjustability, meaning the distribution characteristics adapt to changes during the training phase. The training process of the large-scale mining model based on this embodiment is as follows: Figure 3 As shown.

[0091] Thus, through the above mechanism, this embodiment achieves fine-grained token-level optimization focus. For safety-critical tokens (such as "gas outburst" and "emergency evacuation"), static weights are assigned high base weights. If the model's current prediction is inaccurate, the dynamic weights are further amplified, forming a "double reinforcement." For general descriptive tokens (such as "this work area" and "production process"), static weights are close to neutral, while dynamic weights are adjusted according to contextual relevance. For tokens that have been mastered, dynamic weights are automatically reduced, reallocating optimization resources to content with greater learning value, achieving "automatic course learning."

[0092] In practical applications, after actual verification, the method of this embodiment shows that in terms of training efficiency, the early effective guidance coverage is improved from 31.2% of the standard GRPO to 78.6%, the number of iterations required to reach the target safety response accuracy is reduced from 2,500 steps to 850 steps, and the later optimization focusing accuracy (Top-20% token gradient ratio) is improved from 20% to 35%-42%. In terms of safety recognition performance, the F1-score for major accident risk recognition is improved from 0.73 to 0.91, the recall rate of hidden risk clues is improved from 0.58 to 0.82, and the reward coverage is improved from 67.4% to 89.6%. In terms of fine-grained semantic learning, the token-level safety semantic capture capability is significantly enhanced, and the accuracy of long sequence safety dependency modeling is improved by 27.4% compared to the baseline.

[0093] To implement the above embodiments, this application also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the mine large model reinforcement learning optimization method based on adaptive advantage estimation as described in any of the first aspect embodiments above.

[0094] To implement the above embodiments, this application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the mine large model reinforcement learning optimization method based on adaptive advantage estimation as described in any one of the first aspect embodiments above.

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0099] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A reinforcement learning optimization method for large-scale mining models based on adaptive advantage estimation, characterized in that, Includes the following steps: Multiple candidate response sequences for a mine safety query are obtained, and static weights are assigned to each term in each candidate response sequence based on a pre-built mine safety domain thesaurus. The static weights reflect the risk level of the term in the mine safety domain. Based on the gradient information of the policy model when generating each word, the dynamic weight of each word is calculated. The dynamic weight is adaptively adjusted according to the training stage. In the early stage of training, the gradient norm of the word is emphasized, and in the later stage of training, the contribution of the word is emphasized. Based on the current training stage, the static weights and the dynamic weights are fused to generate fused weights for each word element. In the early stage of training, the fusion coefficient of the static weights is higher than that of the dynamic weights, and in the later stage of training, the fusion coefficient of the dynamic weights is higher than that of the static weights. The relative advantage estimates within a group are weighted based on the fusion weights to generate adaptive advantage estimates for each word, and the parameters of the strategy model are updated using the adaptive advantage estimates.

2. The method according to claim 1, characterized in that, The method of assigning static weights to each term in each candidate response sequence based on a pre-constructed vocabulary for the field of mine safety includes: The Aho-Corasick automaton is used to match each word in the candidate response sequence to query whether each word has a direct entry in the vocabulary of the mining safety field. If it does, the risk level and static weight corresponding to the word are directly obtained. For lexical elements that do not match a direct entry, check whether the lexical element is a prefix of a compound term in the mining safety thesaurus. If so, assign a value according to the prefix importance decay rule. For lexical units that do not match a direct entry and are not prefixes, the semantic similarity between the lexical unit and each entry in the mining safety domain lexicon is calculated using a pre-trained language model encoder, and the lexical units with high similarity are assigned a downgraded extended weight. By combining the local context of each word's location, it is determined whether the word is in a security risk description context, and context-aware weight fine-tuning is performed to output a static weight vector of the same length as the candidate response sequence.

3. The method according to claim 2, characterized in that, The terminology for the field of mine safety adopts a hierarchical organizational structure, with each term labeled with its respective safety subfield, risk level attribute, and typical contextual pattern; The risk levels include Level 1 Major Emergency Risk, Level 2 Significant Risk, Level 3 General Risk, and Level 4 Basic Safety Concept, with the static weights corresponding to each risk level decreasing progressively.

4. The method according to claim 1, characterized in that, The calculation of the dynamic weights of each word based on the gradient information of the policy model when generating each word includes: In the early stages of training, a dynamic weighting strategy prioritizing high sensitivity is adopted. The formula for calculating the dynamic weights is as follows: In the early stages of training, the number of iterations k is less than a preset proportion of the total number of steps. This is the early strengthening coefficient, with a value range of [0.1, 0.2]. and These are the mean and standard deviation of the gradient norm within the batch, respectively; In the later stages of training, a dynamic weighting strategy prioritizing high-contribution factors is adopted. This strategy involves iterations k in the later stages of training that are greater than or equal to a preset proportion of the total number of steps, with a specific enhancement coefficient. The value range is reduced to [0.05, 0.1].

5. The method according to claim 1, characterized in that, The calculation of the dynamic weights of each word based on the gradient information of the policy model when generating each word also includes: Maintain the historical record of gradient norm within the sliding window, calculate the quantile distribution of gradient norm in real time, and set the threshold for dynamic weight calculation to a quantile that decreases as the training process progresses.

6. The method according to claim 1, characterized in that, The fusion coefficient The formula for calculating the fusion coefficient is as follows: (This is a function of training iteration number k.) in, This represents the total number of training steps.

7. The method according to claim 6, characterized in that, The fusion weight The calculation formula is: in, This is an alignment function used to appropriately amplify the effective range of dynamic weights for terms with high static weights. For static weights, For dynamic weights.

8. The method according to claim 1, characterized in that, The adaptive advantage estimate of each word The calculation formula is: in, This is an estimate of the relative advantage within the group. The fusion weight for word elements, This is the mean of the fusion weights of all tokens in the candidate response sequence.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the mine large model reinforcement learning optimization method based on adaptive advantage estimation as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning optimization method for large mining models based on adaptive advantage estimation as described in any one of claims 1-8.

Images