Vocational education large model training method under fine-grained weighted reward mechanism

Abstract

The invention belongs to the field of natural language processing, and particularly relates to the field of large language models, in particular to an education large model training method under a fine-grained weighted reward mechanism. The method comprises the steps of collecting vocational education field data training strategy models and cleaning data; after completing the training of the base large model, training a sentence-level fine-grained reward model by using supervision data, and designing a context-associated scoring mechanism; after scoring model training and large model supervision fine tuning are completed, large model alignment is carried out, and the strategy is updated according to the total reward value. According to the method, the context factor and the dimension interaction factor are introduced, so that sentence scoring is more reasonable and effective, the safety and reliability of a large language model in the vocational education field are improved, and the method has wide application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, specifically to the field of large language models, and in particular to a large educational model training method under a fine-grained weighted reward mechanism. Background Art

[0002] Large language models are a type of natural language processing model trained through deep learning technology, designed to understand and generate human language. These models are usually based on neural network architectures, such as the Transformer architecture, and have hundreds of millions or even hundreds of billions of parameters. They are trained on large-scale corpora to master the rich semantic and grammatical knowledge of the language. With the development of ChatGPT, this technology has been presented to the public in a popular way, and people have also seen its potential value.

[0003] Large language models have broad application prospects in the field of education. By training large language models, we can develop intelligent tutoring systems, automatic homework grading systems, and personalized learning platforms, thereby improving teaching efficiency, reducing the burden on teachers, and providing students with more targeted learning resources. However, since large models are generative results, their security and reliability determine whether they can be used in educational environments. Therefore, how to train a safe and reliable large language model is particularly important. Summary of the invention

[0004] The present invention completes the human feedback reinforcement learning training process through an improved fine-grained weighted reward mechanism, thereby improving the security and reliability of large language models. In the training of large language models in the field of vocational education, the security and reliability of the generated results are crucial. In order to further improve the performance of the model, a complex sentence-level scoring reward model is designed. By considering the contextual relevance between sentences, the scoring mechanism is made more reasonable and effective. A large model training method for vocational education under a fine-grained weighted reward mechanism is proposed.

[0005] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0006] Step 1: First, collect data in the field of vocational education, train a large model, and define it as a strategy model. The collected data in the field of vocational education includes text data such as course materials, PPTs, handouts, exercises, and test papers in the field. The data formats include xlsx, slx, txt, json, docx, md, and pdf. The collected data is uniformly cleaned, including OCR recognition and text extraction of non-text format data, filtering illegal characters, and finally obtaining clean text data;

[0007] Step 2: After completing the training of the base model, perform supervised fine-tuning of the model. Use supervised data to train a sentence-level fine-grained reward model and design a context-related scoring mechanism. By introducing context factors, the score of a sentence depends not only on its own characteristics, but also on the influence of the sentences before and after it.

[0008] Step 3: After completing the scoring model training and large model supervision fine-tuning, align the large model and update the strategy based on the total reward value.

[0009] Preferably, in step 2, supervised data is used to train a sentence-level fine-grained reward model, and the specific implementation steps of designing a context-related scoring mechanism are as follows:

[0010] Step 1: Introduce context factor C i , which is used to measure the relevance of the i-th sentence to its context. The context factor formula is where γ i-1,i Indicates the relevance of the i-th sentence to the previous sentence, γ i,i+1 It represents the relevance between the i-th sentence and the next sentence. The relevance γ is calculated based on the semantic similarity method, and its calculation formula is γ i,j =sim(S i ,S j ), where sim(S i ,S j ) represents the semantic similarity between the i-th sentence and the j-th sentence, calculated by the cosine similarity represented by the vector where v i , v j are the vector representations of the i-th sentence and the j-th sentence respectively;

[0011] Step 2: Study the dimension interaction factors. The score of each sentence includes five dimensions: safety S safety , reliability reliability , semantic accuracy S semantic , Fluency S fluency and contextual consistency S context , introducing the dimension interaction factor I ij To indicate that the j-th scoring dimension of the i-th sentence is affected by other dimensions, the calculation formula of the dimension factor is Among them I ij Indicates that the j-th scoring dimension of the i-th sentence is affected by other dimensions, α jk represents the influence weight of the jth dimension on the kth dimension, S ik represents the score of the kth dimension of the i-th sentence;

[0012] Step 3: Final score S′ for each sentence i Combined with its own rating S i, context factor C i And the dimension interaction factor is calculated, and its formula is: Among them, λ1, λ2, and λ3 are adjustment parameters that control the weights of self-rating, contextual factors, and dimension interaction factors, satisfying λ1+λ2+λ3=1, β j Represents the weight of the j-th scoring dimension.

[0013] As a preferred embodiment, the large model alignment in step 3 is specifically implemented according to the total reward value update strategy. If the model generates n sentences at time step t, the total reward R t The sum of the final scores for these sentences The model updates the strategy based on the total reward value. The strategy is updated using the policy gradient method, and its update formula is: Where θ is the model parameter, α is the learning rate, and J(θ) is the objective function, which is defined as According to the policy gradient theorem, the gradient of the objective function is where a t For state s t The action selected is the generated sentence, R t is the reward value of this action.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are that a complex sentence-level scoring reward model with context association and dimension interaction is proposed. By introducing context factors and dimension interaction factors, the scores between sentences and between scoring dimensions influence each other, thereby improving the overall text quality. The designed fine-grained weighted reward mechanism can more reasonably and effectively evaluate the quality of sentences, taking into account the sentence's own characteristics, context association, and the mutual influence between multiple scoring dimensions. In the reinforcement learning framework, using this scoring mechanism to train the model can effectively improve the application effect of the large language model in the field of vocational education, making the results it generates safer and more reliable. DETAILED DESCRIPTION

[0015] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0016] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0017] Embodiment, the present invention proposes a complex sentence-level scoring reward model with context association and dimension interaction, which introduces context factors and dimension interaction factors to make the scores between sentences and between scoring dimensions influence each other, thereby improving the overall text quality. In the reinforcement learning framework, using this scoring mechanism to train the model can effectively improve the application effect of large language models in the field of vocational education.

[0018] In order to effectively improve the effect of large models in vertical fields, this method starts with the training of the base model. First, data in the field of vocational education is collected, a large model is trained, and it is defined as a strategy model.

[0019] The collected vocational education data includes text data such as course materials, PPT, handouts, exercises, and test papers in the field, and the data formats include xlsx, slx, txt, json, docx, md, pdf, etc. The collected data is cleaned uniformly, including OCR recognition of non-text format data to extract text, filtering illegal characters, etc. Finally, clean text data is obtained. After completing the data collection, similar to the conventional large language model, we choose a model training strategy model based on the Transformer decoder.

[0020] After completing the training of the base model, use the supervised data to train the sentence-level fine-grained reward model and design a context-related scoring mechanism. By introducing the context factor, the score of the sentence depends not only on its own characteristics, but also on the influence of the sentences before and after it. The specific implementation steps are to introduce the context factor C i , which is used to measure the relevance of the i-th sentence to its context. The context factor formula is where γ i-1,i Indicates the relevance of the i-th sentence to the previous sentence, γ i,i+1 It represents the relevance between the i-th sentence and the next sentence. The relevance γ is calculated based on the semantic similarity method, and its calculation formula is γ i,j =sim(S i ,S j ), where sim(S i ,S j ) represents the semantic similarity between the i-th sentence and the j-th sentence, calculated by the cosine similarity represented by the vector where v i , v j are the vector representations of the i-th sentence and the j-th sentence respectively; the dimension interaction factor is studied, and the score of each sentence includes five dimensions: security S safety , reliability reliability , semantic accuracy S semantic , Fluency S fluency and contextual consistency S context , introducing the dimension interaction factor Iij To indicate that the j-th scoring dimension of the i-th sentence is affected by other dimensions, the calculation formula of the dimension factor is Among them I ij Indicates that the j-th scoring dimension of the i-th sentence is affected by other dimensions, α jk represents the influence weight of the jth dimension on the kth dimension, S ik represents the score of the kth dimension of the i-th sentence; the final score of each sentence S′ i Combined with its own rating S i , context factor C i And the dimension interaction factor is calculated, and its formula is: Among them, λ1, λ2, and λ3 are adjustment parameters that control the weights of self-rating, contextual factors, and dimension interaction factors, satisfying λ1+λ2+λ3=1, β j Represents the weight of the j-th scoring dimension.

[0021] After completing the scoring model training, we continue to implement reinforcement learning training based on human weighted feedback. We use the scoring mechanism introduced in the second part to score each sentence generated by the model, and calculate the final score of each sentence by combining the context factor and the dimension interaction factor. If the model generates n sentences at time step t, the total reward R t The sum of the final scores for these sentences The model updates the strategy based on the total reward value. The strategy is updated using the policy gradient method, and its update formula is: Where θ is the model parameter, α is the learning rate, and J(θ) is the objective function, which is defined as According to the policy gradient theorem, the gradient of the objective function is where a t For state s t The action selected is the generated sentence, R t is the reward value of this action.

[0022] In summary, this paper proposes a complex sentence-level scoring reward model with context association and dimension interaction. By introducing context factors and dimension interaction factors, the scores between sentences and between scoring dimensions influence each other, thereby improving the overall text quality. In the reinforcement learning framework, using this scoring mechanism to train the model can effectively improve the application effect of large language models in the field of vocational education.

[0023] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large-scale model training method for vocational education under a fine-grained weighted reward mechanism, characterized in that: The following steps are involved: Step 1: First, collect data in the field of vocational education, train a large model, and define it as a strategy model. The collected data in the field of vocational education includes text data such as course materials, PPTs, handouts, exercises, and test papers in the field. The data formats include xlsx, slx, txt, json, docx, md, and pdf. The collected data is uniformly cleaned, including OCR recognition and text extraction of non-text format data, filtering illegal characters, and finally obtaining clean text data; Step 2: After completing the training of the base model, perform supervised fine-tuning of the model. Use supervised data to train a sentence-level fine-grained reward model and design a context-related scoring mechanism. By introducing context factors, the score of a sentence depends not only on its own characteristics, but also on the influence of the sentences before and after it. Step 3: After completing the scoring model training and large model supervision fine-tuning, align the large model and update the strategy based on the total reward value.

2. According to the method for training a large model of vocational education under a fine-grained weighted reward mechanism according to claim 1, it is characterized in that: In step 2, supervised data is used to train a sentence-level fine-grained reward model, and the specific implementation steps of designing a context-related scoring mechanism are as follows: Step 1: Introduce context factor C i , which is used to measure the relevance of the i-th sentence to its context. The context factor formula is where γ i-1,i Indicates the relevance of the i-th sentence to the previous sentence, γ i,i+1 It represents the relevance between the i-th sentence and the next sentence. The relevance γ is calculated based on the semantic similarity method, and its calculation formula is γ i,j =sim(S i ,S j ), where sim(S i ,S j ) represents the semantic similarity between the i-th sentence and the j-th sentence, calculated by the cosine similarity represented by the vector where v i , v j are the vector representations of the i-th sentence and the j-th sentence respectively; Step 2: Study the dimension interaction factors. The score of each sentence includes five dimensions: safety S safety , reliability reliability , semantic accuracy S semantic , Fluency S fluency and contextual consistency S context , introducing the dimension interaction factor I ij To indicate that the j-th scoring dimension of the i-th sentence is affected by other dimensions, the calculation formula of the dimension factor is Among them I ij Indicates that the j-th scoring dimension of the i-th sentence is affected by other dimensions, α jk represents the influence weight of the jth dimension on the kth dimension, S ik represents the score of the kth dimension of the i-th sentence; Step 3: Final score S′ for each sentence i Combined with its own rating S i , context factor C i And the dimension interaction factor is calculated, and its formula is: Among them, λ1, λ2, and λ3 are adjustment parameters that control the weights of self-rating, contextual factors, and dimension interaction factors, satisfying λ1+λ2+λ3=1, β j Represents the weight of the j-th scoring dimension.

3. The vocational education large model training method under a fine-grained weighted reward mechanism according to claim 1 is characterized in that: The specific implementation of the large model alignment in step 3 and the update strategy based on the total reward value is that the model generates n sentences at time step t, then the total reward R t The sum of the final scores for these sentences The model updates the strategy based on the total reward value. The strategy is updated using the policy gradient method, and its update formula is: Where θ is the model parameter, α is the learning rate, and J(θ) is the objective function, which is defined as According to the policy gradient theorem, the gradient of the objective function is where a t For state s t The action selected is the generated sentence, R t is the reward value of this action.

Images