A Chinese reasoning task model construction method, device, equipment and medium

Abstract

The present application discloses a method, apparatus, device, and medium for constructing a Chinese reasoning task model, which relates to the field of natural language processing, including: using a Chinese dialogue dataset to perform model training on a low-order adaptive model and a large language model respectively, to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training; using a Chinese reasoning training dataset to perform secondary training on the initial Chinese understanding task model, to obtain a Chinese reasoning task model after secondary training; performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training, to obtain the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training, and using the relative entropy to perform model update to obtain the currently updated Chinese reasoning task model. This effectively optimizes parameter efficiency, reduces resource requirements, and improves the performance of the Chinese reasoning task model.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method, device, equipment and medium for constructing a Chinese reasoning task model. Background Art

[0002] The release of GPT-3.5 (Generative Pre-trained Transformer) has sparked widespread excitement, marking a major advancement in the field of LLMs (Large Language Models). LLMs have proven highly effective in a variety of natural language processing tasks, such as text generation and reasoning. All of these tasks follow simple instructions. Several recent studies have aimed to transform LLMs into models that can accurately follow instructions. Choosing appropriate instructions is crucial for NLP (Neuro-Linguistic Programming) tasks as they directly impact model performance. Instructions need to be tailored to the specific task. On the other hand, it has been observed that increasing the number of model parameters does not always lead to consistent improvements in performance. In fact, smaller models perform well on certain specific NLP tasks, highlighting the importance of understanding model structure and using appropriate training methods.

[0003] The large language models that currently exist are mainly targeted at English datasets, which means they may not be effective enough in understanding and reasoning about Chinese. This problem is caused by the imbalance of training data and the limited adaptability of the model structure. A model called LLaMA (Large Language Model Meta AI) was recently developed. This model has been extensively trained using billions of samples and is known for its outstanding competitive performance. Compared with traditional language models, the LLaMA model has many advantages. It performs better in tasks that require drawing conclusions because it is trained with a more comprehensive vocabulary range. According to the evaluation of the LLaMA-13B model, it performs well in most benchmarks, surpassing the current GPT-3.5 model. However, large models still have some problems with high demands on memory and storage resources.

[0004] In summary, how to build a Chinese reasoning task model to effectively improve the ability to understand and reason about Chinese tasks, while reducing the high demand for memory and storage resources during training of the Chinese reasoning task model is a technical problem to be solved in this field. Summary of the Invention

[0005] In view of this, the present invention aims to provide a method, apparatus, device, and medium for constructing a Chinese reasoning task model, capable of constructing a Chinese reasoning task model that effectively improves the ability to understand and reason about Chinese tasks, while also reducing the high memory and storage resource requirements of the Chinese reasoning task model during training. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a method for constructing a Chinese reasoning task model, comprising:

[0007] The low-level adaptive model and the large language model are trained using the Chinese dialogue dataset to obtain an initial Chinese understanding task model constructed by the low-level adaptive model and the large language model after initial training.

[0008] Performing secondary training on the initial Chinese comprehension task model using a Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model;

[0009] Human feedback reinforcement learning is performed on the Chinese reasoning task model after secondary training, and the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training is obtained. The relative entropy is used to update the model to obtain the current updated Chinese reasoning task model.

[0010] Optionally, the method of using the Chinese dialogue dataset to perform model training on the low-order adaptive model and the large language model respectively to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training includes:

[0011] Obtain a Chinese dialogue dataset, and perform encoding and decoding on the Chinese dialogue dataset to obtain Chinese dialogue sentence vectors;

[0012] The low-rank matrix of the low-order adaptive model is used to freeze the model weight parameters of the large language model to obtain a low-rank fine-tuned large language model;

[0013] Using the Chinese dialogue sentence vectors to train a low-order adaptive model and a low-rank fine-tuned large language model to obtain a low-order adaptive model and a large language model after initial training;

[0014] Merging the output result vectors of the low-order adaptive model and the large language model after the initial training to obtain an understanding skill matrix representing the degree of understanding of the Chinese conversation intent;

[0015] An initial Chinese comprehension task model including the comprehension skill matrix is ​​constructed based on the low-order adaptive model and the large language model after the initial training.

[0016] Optionally, the performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model includes:

[0017] Obtain a Chinese reasoning training data set, and encode and decode the Chinese reasoning training data set to obtain a Chinese reasoning sentence vector;

[0018] Using the Chinese reasoning sentence vectors, the thinking reasoning of the initial Chinese comprehension task model is trained to obtain a reasoning Chinese comprehension task model including a reasoning ability skill matrix; the reasoning ability skill matrix represents the degree of reasoning about the context of the Chinese sentence dialogue;

[0019] The reasoning ability skill matrix in the Chinese comprehension task model is connected with the understanding skill matrix in the initial Chinese comprehension task model to generate a secondary trained Chinese reasoning task model including the understanding reasoning feature skill matrix.

[0020] Optionally, after performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model, the method further includes:

[0021] Inputting a Chinese test question into the second-trained Chinese reasoning task model, so that the second-trained Chinese reasoning task model determines a corresponding target instance in a thought chain instance according to the question type of the Chinese test question;

[0022] The target instance is used to generate a corresponding reasoning train of thought, so as to generate a Chinese test answer corresponding to the Chinese test question based on the reasoning train of thought.

[0023] Optionally, performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training includes:

[0024] The Chinese reasoning task model after secondary training that outputs the Chinese test answer is subjected to human feedback reinforcement learning, the answer matrix constructed by the Chinese test answer is scored, and the Chinese test answer with the highest score is selected from all scores and output as the final output result.

[0025] Optionally, before inputting the Chinese test question into the secondarily trained Chinese reasoning task model so that the secondarily trained Chinese reasoning task model determines the corresponding target instance in the thought chain instance according to the question type of the Chinese test question, the method further includes:

[0026] Using a preset clustering algorithm and dividing the Chinese question training samples into different numbers of Chinese question training clusters according to question types;

[0027] Sort the Chinese question training samples in each Chinese question training cluster in ascending order according to their distance from the cluster center to create a Chinese question matrix table;

[0028] A sample matrix table including a question matrix and an answer matrix is ​​generated based on answer training samples and question training samples in the Chinese question training samples and normalized.

[0029] Optionally, the using the relative entropy to perform model updating to obtain a currently updated Chinese reasoning task model includes:

[0030] The relative entropy is set to a negative value, and the relative entropy is added to the reward signal of the reward model to update the reward model, and a loop operation of human feedback reinforcement learning is performed on the Chinese reasoning task model after secondary training using the updated reward model until the number of cycles reaches a preset threshold, and the updated Chinese reasoning task model is output.

[0031] In a second aspect, the present application discloses a Chinese reasoning task model construction device, comprising:

[0032] A model building and training module is used to train the low-level adaptive model and the large language model using the Chinese dialogue dataset to obtain an initial Chinese understanding task model constructed by the low-level adaptive model and the large language model after initial training;

[0033] A model training module, configured to perform secondary training on the initial Chinese comprehension task model using a Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model;

[0034] A model update module is used to perform human feedback reinforcement learning on the Chinese reasoning task model after the second training, and obtain the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after the second training, and use the relative entropy to update the model to obtain the current updated Chinese reasoning task model.

[0035] In a third aspect, the present application discloses an electronic device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor is used to execute the computer program to implement the steps of the aforementioned disclosed method for constructing a Chinese reasoning task model.

[0038] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed method for constructing a Chinese reasoning task model are implemented.

[0039] It can be seen that the present application discloses a method for constructing a Chinese reasoning task model, comprising: using a Chinese dialogue dataset to perform model training on a low-order adaptive model and a large language model respectively, so as to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training; using a Chinese reasoning training dataset to perform secondary training on the initial Chinese understanding task model to obtain a Chinese reasoning task model after secondary training; performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training, and obtaining the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training, and using the relative entropy to update the model to obtain the current updated Chinese reasoning task model. It can be seen that the Chinese reasoning task model constructed by the low-order adaptive model and the large language model ensures that the large language model maintains excellent task performance while combining the low-rank fine-tuning of the large language model by the low-order adaptive model, effectively optimizes parameter efficiency, reduces resource requirements, and solves the inherent limitations of traditional reinforcement learning by incorporating human feedback into the training process. The introduction of human feedback significantly enhances the training process, thereby improving the performance of the Chinese reasoning task model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 A flowchart of a method for constructing a Chinese reasoning task model disclosed in this application;

[0042] Figure 2 This is a flow chart of a Chinese reasoning task model training method disclosed in this application;

[0043] Figure 3 A flowchart of a specific Chinese reasoning task model construction method disclosed in this application;

[0044] Figure 4 This is a pseudo code diagram for constructing a thought chain example disclosed in this application;

[0045] Figure 5 A pseudo code diagram of the final output result process disclosed in this application;

[0046] Figure 6 A schematic diagram of the structure of a Chinese reasoning task model construction device disclosed in this application;

[0047] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The release of GPT-3.5 has sparked widespread excitement, representing a major advancement in the field of LLMs. LLMs have proven to be very effective in a variety of natural language processing tasks, such as text generation and reasoning. All of these tasks follow simple instructions. Several recent studies have aimed to transform LLMs into models that can accurately follow instructions. Choosing appropriate instructions is crucial for NLP tasks as they directly impact model performance. Instructions need to be tailored to the specific task. On the other hand, it has been observed that increasing the number of parameters in a model does not always lead to consistent improvements in its performance. In fact, smaller models perform satisfactorily on certain specific NLP tasks, highlighting the importance of understanding the model structure and using appropriate training methods.

[0050] Existing large language models primarily target English datasets, which means they may not be effective at understanding and reasoning about Chinese. This problem is caused by imbalanced training data and the limited adaptability of the model architecture. A model called LLaMA was recently developed. This model has been extensively trained using billions of samples and is known for its outstanding competitive performance. Compared to traditional language models, the LLaMA model offers many advantages. It performs better in tasks that require conclusion drawing because it is trained using a more comprehensive vocabulary. Evaluations of the LLaMA-13B model show excellent performance on most benchmarks, surpassing the current GPT-3.5 model. However, large models still have some challenges in terms of high memory and storage requirements.

[0051] To this end, this application provides a Chinese reasoning task model construction solution, which can construct a Chinese reasoning task model to effectively improve the understanding and reasoning ability of Chinese tasks, while reducing the high demand for memory and storage resources of the Chinese reasoning task model during training.

[0052] Reference Figure 1 As shown, an embodiment of the present invention discloses a method for constructing a Chinese reasoning task model, comprising:

[0053] Step S11: Use the Chinese dialogue dataset to perform model training on the low-order adaptive model and the large language model respectively to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training.

[0054] In this embodiment, a Chinese conversation dataset is obtained and encoded and decoded to obtain Chinese conversation sentence vectors. The model weight parameters of the large language model are frozen using the low-rank matrix of the low-rank adaptive model to obtain a low-rank fine-tuned large language model. The low-rank adaptive model and the low-rank fine-tuned large language model are trained using the Chinese conversation sentence vectors to obtain the low-rank adaptive model and the large language model after initial training. The output vectors of the low-rank adaptive model and the large language model after initial training are combined to obtain an understanding skill matrix representing the degree of understanding of the Chinese conversation intent. An initial Chinese comprehension task model containing the understanding skill matrix is ​​constructed based on the low-rank adaptive model and the large language model after initial training. It can be understood that the Chinese conversation dataset is obtained as input data, the input data undergoes an initial encoding and decoding process, the information is converted into vector form, and then input into the low-rank adaptive model LoRA (Low-Rank Adaptation of Large Language Models) and the large language model LLaMA for training. During the LLaMA training process, the relevant parameters of LLaMA remain unchanged, and the LoRA low-rank fine-tuning is also used. This enables the LLaMA model to be trained simultaneously with LoRA, preserving the acquired skills in the new model. Specifically, the first phase of training aims to enhance the proficiency of the Chinese understanding task model in Chinese understanding. Training is performed using multiple Chinese dialogue datasets, wherein the Chinese dialogue datasets specifically include: Chinese dialogue data, Alapca-data of Chinese translation, and other related datasets. During the initial training process, sentence encoding and decoding technology is applied to generate a sentence matrix table, which is used to represent the Chinese dialogue sentence vector matrix formed by the Chinese dialogue data and Alapca-data after encoding and decoding processing, specifically expressed as: in, Represents the Chinese conversation sentence vector of the kth Chinese conversation data or Chinese translated Alapca-data. The large language model LLaMA and the LoRA model are trained together as a combined model. Throughout the training phase, the parameters of the large language model remain fixed, while low-rank matrix factorization is applied to the input of the LoRA model. Therefore, by combining the outputs of the large language model and the LoRA model, the final understanding skill matrix is ​​obtained, which is expressed as: in, Represents the kth comprehension skill matrix element resulting from the combined output of the LoRA model and the LLaMA model. These comprehension skill matrices are then stored in the initial Chinese comprehension task model, completing the initial fine-tuning step of the LoRA model. During the training process, the fine-tuning of the LoRA model is mathematically expressed as: W = W plm +λW LoRA =W plm +λE LoRAzeros ×F LoRAgaussian ;W represents the model weight of the model, λ represents the weight of the hyperparameter of the LoRA model, plm represents the weight of the pre-trained model LLaMA, E LoRAzeros Represents the LoRA model zero matrix, F LoRAgaussian represents the LoRA model Gaussian matrix. By adjusting λ, the need for hyperparameter tuning of the LoRA model is reduced. By incorporating the LoRA-fine-tuned LLaMA model into the initial Chinese language understanding task model, the efficiency and performance of LLM are significantly improved. The key goal is to strike a balance between reducing the number of trainable parameters and ensuring optimal task performance. By reducing the number of parameters, the goal is to improve the computational efficiency and memory requirements of LLM, making it more practical in real-world applications.

[0055] Step S12: performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model.

[0056] In this embodiment, a Chinese reasoning training data set is obtained, and the Chinese reasoning training data set is encoded and decoded to obtain a Chinese reasoning sentence vector; the Chinese reasoning sentence vector is used to train the thinking reasoning of the initial Chinese comprehension task model to obtain a reasoning Chinese comprehension task model including a reasoning ability skill matrix; the reasoning ability skill matrix represents the degree of reasoning of the Chinese sentence context dialogue; the reasoning ability skill matrix in the reasoning Chinese comprehension task model is connected with the understanding skill matrix in the initial Chinese comprehension task model to generate a secondary trained Chinese reasoning task model including an understanding reasoning feature skill matrix. It can be understood that using CoT (Chain of Thoughts) training data as a Chinese reasoning training data set to develop the reasoning ability of the initial Chinese comprehension task model is a secondary training process. The second training stage involves incorporating the CoT training data into the initial Chinese comprehension task model trained in the first step to enhance its reasoning ability. The training process of this stage is similar to the first step. The CoT training data generates a sentence matrix through the sentence encoding and decoding layer. That is, the sentence matrix constructed by the Chinese reasoning sentence vector. The initial Chinese comprehension task model is trained using the LLaMA model and the LoRA model to obtain the final reasoning ability skill matrix in, represents the kth element of the reasoning ability skill matrix, which is the result of combining the output of the LoRA model and the LLaMA model trained with the CoT training data. Then, the reasoning ability skill matrix is ​​stored in the currently trained Chinese comprehension task model, completing the second step of the LoRA fine-tuning process. Finally, the reasoning ability skill matrix is ​​concatenated with the comprehension skill matrix to form Q skill Understand the reasoning feature skill matrix. The connection process is expressed as: Q skill =[J chinese ; J CoT ]. It is important to note that combining LLM with the Chain of Thought (CoT) method to enhance the reasoning ability of Chinese comprehension task models has the potential to reduce training costs and dataset dependence. In this way, the comprehensive Chinese CoT data corpus is utilized to enhance the understanding and reasoning capabilities of Chinese reasoning task models. Training on this specific dataset enables Chinese reasoning task models to gain a deeper understanding of Chinese language tasks, thereby improving performance and effectively addressing the challenges faced by Chinese text understanding and reasoning.

[0057] Step S13: Perform human feedback reinforcement learning on the Chinese reasoning task model after secondary training, and obtain the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training, and use the relative entropy to update the model to obtain the current updated Chinese reasoning task model.

[0058] In this embodiment, the relative entropy is set to a negative value, and the relative entropy is added to the reward signal of the reward model to update the reward model, and a cyclic operation of performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training using the updated reward model is performed until the number of cycles reaches a preset threshold, and the updated Chinese reasoning task model is output. It can be understood that the reward model is used to optimize the Chinese reasoning task model after secondary training. Specifically, the relative entropy between the output result of the Chinese reasoning task model after secondary training and the output result trained by reinforcement learning using the reward model, that is, the Kullback-Leibler divergence, is incorporated into the reward signal. By assigning a negative value to the relative entropy, the Chinese understanding task model is prevented from deviating excessively from its original output. Therefore, the final reward calculation formula for the RLHF (Reinforcement Learning from Human Feedback, human feedback reinforcement learning) model, that is, the progressive update of the reward model is r=r θ -λr KL ; Among them, r θ represents the standard reward, λ represents the weight randomly generated by the reward model, r KL represents the scaling of the divergence. r θ 、rKL It can reinforce the learning strategy to generate results that significantly deviate from the untrained model in each batch, and introduce the latest technology of human feedback reinforcement learning RLHF technology into the Chinese reasoning task model, further improving the Chinese reasoning task model's understanding and reasoning ability of the Chinese language, and improving the Chinese reasoning task model's understanding of human preferences. Through the above-mentioned initial training, secondary training, and human feedback reinforcement, the Chinese reasoning task model is continuously trained and updated, and finally the updated Chinese reasoning task model LLaMA-LoRA-RLHF is obtained. The introduction of the LoRA model significantly supports reducing the required resources. The LoRA model effectively reduces the consumption of parameter storage and virtual random access memory VRAM (Virtual Random Access Memory). It is worth noting that when the dimension is much smaller than the model, the LoRA model eliminates the need to store the optimizer state parameters, thereby significantly reducing the usage of VRAM.

[0059] Reference Figure 2As shown, an embodiment of the present invention also discloses a method for constructing and training a Chinese reasoning task model. The specific process includes: first, encoding the Chinese question-and-answer data in a Chinese dialogue dataset using a key-value pair QA method, then forming a Chinese dialogue sentence vector matrix from the encoded Chinese sentences, and finally decoding the Chinese dialogue sentence vector matrix to generate a dataset for training the understanding capabilities of the low-level adaptive model (LoRA) and the large-scale language model (LLaMA). The Chinese dialogue sentence vector matrix is ​​input into the LoRA model and the LLaMA model, respectively. At this time, the weight parameters of the LLaMA model are frozen, meaning they do not participate in training. However, the LoRA model is trained by adding additional network layers to the LoRA model and only training the parameters of these newly added network layers. Because these additional parameters are relatively small, not only is the cost of fine-tuning significantly reduced, but similar results to full model fine-tuning can be achieved. Through model fine-tuning training of the LoRA model, a bypass is added to the original PLM of the LLaMA model, performing a dimensionality reduction and then dimensionality increase operation to simulate the so-called intrinsic dimensionality. During training, the parameters of the PLM are fixed, and only the reduced dimensionality matrix A and the increased dimensionality matrix B are trained. The model's input and output dimensions remain unchanged, and the parameters of the BA and plm are superimposed at the output. A is initialized with a random Gaussian distribution, and B is initialized with a zero matrix to ensure that the bypass matrix remains zero at the start of training. A follows a normal distribution, and B is a zero matrix. The initial Chinese comprehension task model is then obtained. Next, the initial Chinese comprehension task model is retrained. This training uses a Chinese reasoning dataset. The Chinese reasoning data is sequentially encoded using a key-value pair QA method. The encoded Chinese reasoning sentences are then converted into a Chinese reasoning sentence vector matrix. Finally, the Chinese reasoning sentence vector matrix is ​​decoded to generate a dataset used to train the reasoning capabilities of the initial Chinese comprehension task model. After training on the Chinese reasoning dataset, a retrained Chinese reasoning task model is obtained. Note that this retrained Chinese reasoning task model is the initial Chinese comprehension task model with reasoning capabilities. The retrained Chinese reasoning task model is then reinforced using a reward model to obtain an updated Chinese reasoning task model.

[0060] It can be seen that the present application discloses a method for constructing a Chinese reasoning task model, comprising: using a Chinese dialogue dataset to perform model training on a low-order adaptive model and a large language model respectively, so as to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training; using a Chinese reasoning training dataset to perform secondary training on the initial Chinese understanding task model to obtain a Chinese reasoning task model after secondary training; performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training, and obtaining the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training, and using the relative entropy to update the model to obtain the current updated Chinese reasoning task model. It can be seen that the Chinese reasoning task model constructed by the low-order adaptive model and the large language model ensures that the large language model maintains excellent task performance while combining the low-rank fine-tuning of the large language model by the low-order adaptive model, effectively optimizes parameter efficiency, reduces resource requirements, and solves the inherent limitations of traditional reinforcement learning by incorporating human feedback into the training process. The introduction of human feedback significantly enhances the training process, thereby improving the performance of the Chinese reasoning task model.

[0061] Reference Figure 3 As shown, the embodiment of the present invention discloses a specific method for constructing a Chinese comprehension task model. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically:

[0062] Step S21: Use the Chinese dialogue dataset to perform model training on the low-order adaptive model and the large language model respectively to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training.

[0063] Step S22: performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model.

[0064] For more detailed processing procedures in steps S21 and S22, please refer to the aforementioned disclosed embodiments, which will not be described again here.

[0065] Step S23: Input the Chinese test question into the Chinese reasoning task model after secondary training, so that the Chinese reasoning task model after secondary training determines the corresponding target instance in the thinking chain instance according to the question type of the Chinese test question; and uses the target instance to generate the corresponding reasoning ideas, so as to generate the Chinese test answer corresponding to the Chinese test question based on the reasoning ideas.

[0066] In this embodiment, during the question evaluation phase, Chinese test questions are fed into the retrained Chinese reasoning task model, and corresponding related questions in thought chain instances are identified based on the question type. These thought chain instances provide valuable guidance for generating answers to the test questions, promoting the formation of higher-quality reasoning processes. This allows human feedback experts to evaluate the quality of the generated answers during the answer generation process.

[0067] In this embodiment, before inputting Chinese test questions into the retrained Chinese reasoning task model so that the retrained Chinese reasoning task model determines the corresponding target instance in the thought chain instance based on the question type of the Chinese test question, the method further includes: generating thought instances of the thought chain. Specifically, during the thought chain instance generation stage, diversity cluster analysis is used to implement iterative problem segmentation using K-means clustering. This ensures that different Chinese question training samples are assigned to different groups, each group containing multiple Chinese question training samples. The introduction of the diversity cluster analysis method is of great significance for improving the performance and accuracy of the LLM in logical reasoning tasks, while also providing effective guidance and training examples for generating more coherent and accurate thought chains. These Chinese question training samples are then processed to generate fixed-size vector representations. Subsequently, diverse Chinese question instances are extracted from different groups. In addition, the instance generation process also incorporates guidance from human feedback reinforcement learning. Expert feedback is provided on the logical process of thought chain formation to ensure that the content is consistent with human expectations.

[0068] Among them, in the instance generation stage of the thinking chain, diversity cluster analysis is used to realize iterative problem segmentation using K-means clustering, which specifically includes: using a preset clustering algorithm and dividing the Chinese problem training samples into different numbers of Chinese problem training clusters according to the problem type; sorting the Chinese problem training samples in each Chinese problem training cluster in ascending order according to the distance from the cluster center to create a Chinese problem matrix table; creating and normalizing the answer training samples and question training samples in the Chinese problem training samples to generate a sample matrix table containing a question matrix and an answer matrix. It can be understood that the K-means (K-means clustering algorithm) clustering algorithm is applied to divide the Chinese problem training samples into n groups, and effectively group different types of Chinese problem training samples into different clusters. Each cluster contains multiple instance problems. Within each cluster, the problems can be sorted in ascending order according to the distance between the problem and the cluster center. For example: given a group problem t, a Chinese problem matrix table can be created, expressed as: in, Represents the Chinese question matrix elements. The Chinese question matrix table should be arranged in ascending order, and the value of t should be between 1 and n. The model then combines the input question with the corresponding related answer steps to create the final question answer sample matrix table S = [s1 , s 2 ,...,s n ]. For each sample s i A normalization analysis was performed, where i∈[1, 2, 3, ..., n], to ensure that the generated sample matrix had uniform dimensionality. The question-answer sample matrix consisted of a Chinese test question matrix and a Chinese test answer matrix. The Chinese answer matrix was formed by concatenating the inference matrix and the final Chinese answer matrix. Figure 4 A comprehensive process for generating sample thought chains for the thought chain described in pseudocode 1.

[0069] Step S24: Performing human feedback reinforcement learning on the secondary trained Chinese reasoning task model that outputs the Chinese test answer, scoring the answer matrix constructed by the Chinese test answer, and selecting the Chinese test answer with the highest score from all scores as the final output result.

[0070] In this embodiment, a Chinese test question is received as input, and encoding and decoding operations are performed to generate a Chinese test question matrix Q test In the result construction phase, the generated question answer sample matrix S = [s 1 , s 2 ,...,s n ] is taken into consideration. Final Chinese test answer matrix Q result Based on the results in the question answer sample matrix table s k ∈S is constructed step by step. When generating the Chinese test answer matrix, the reward model assigns a score to the corresponding answer matrix and selects the output result with the highest score in the thinking chain as the final output result. This selection improves the Chinese test answer Q result The quality of Q result By the inference process matrix r result And the final result matrix a result Merged. For the final output process, refer to Figure 4 Pseudo code 2 is shown. Then, the evaluation method of 526RCCA Reviews is used to measure the performance of the multi-round Chinese dialogue system. The optimization goal presented by the following formula aims to ensure that any set of typical variables h i and q j The specific formula is as follows:

[0071]

[0072]

[0073] in, represents the transpose of the eigenvector in the projection matrix, X represents the first sentence in a multi-round dialogue sample, Z represents the last sentence in a multi-round dialogue sample, and q j Denotes the projection vector of auxiliary Z, C XX represents the covariance between X and X, represents the transpose of the projection vector, C zz represents the covariance between Z and Z, h i Represents the eigenvector in the projection matrix.

[0074] When the number of observed Chinese test questions is small, the constraint term r is included in the covariance matrix. X and r Z > 0, which can ensure the convergence of the above optimization objectives. The formula is as follows:

[0075]

[0076]

[0077] Where E represents the identity matrix, Represented by the constraint factor and the identity matrix and C XX The covariance between the determined X and X, r X Indicated Constraint factor, Represents the correlation covariance between Z and Z determined by the constraint factor and the identity matrix, r Z express Constraint factor. It can help optimize the objective function to converge faster.

[0078] The projection matrices H and Q can be obtained by generalized eigenvalue decomposition, as follows:

[0079]

[0080] The feature representations of sentences in the latent vector space W are represented as W X =H T X and W Z =Q T Z. The distance between these two vectors is calculated and analyzed using the following formula:

[0081]

[0082] in, Represents the correlation of X in W after projection mapping, Represents the correlation of Z in W after projection mapping.

[0083] When the Chinese reasoning task model is built, the model is evaluated for performance. The main evaluation indicators used in the performance evaluation may include but are not limited to: accuracy, CPT4, perplexity and manual evaluation. Specifically, the accuracy Acc (Accuracy): measures the proportion of correct predictions made by the model on a given input. The GPT-4: different instructions will be provided to GPT-4 to evaluate its performance on various test tasks. It should be noted that no research has yet proven the superiority of GPT-4. Therefore, it is generally only required to score GPT-4 based on its fluency. The manual evaluation (Human): is an evaluation method for comprehensively evaluating the model's language fluency, answer relevance and other indicators when dealing with open-ended questions and answers. It is evaluated through a manual scoring mechanism with a full score of 100. The perplexity PPL (Perplexity): evaluates the model's language modeling ability. It quantifies the difference between the predicted results and the actual results. A lower value indicates better performance. The expression is:

[0084] PPL=p(w1,w2,w3……w n ) (-1 / m) ;

[0085] Among them, w n Represents the difference between the nth predicted result and the actual result.

[0086] After a comprehensive evaluation of the model, it was compared with eight other Chinese models through experimental analysis. Specifically, the results were presented on benchmark tasks in seven fields, covering a wide range of task areas. These fields include basic natural language understanding and generation, as well as the application of natural language reasoning capabilities, such as knowledge question answering, multi-round dialogue understanding, and Chinese logical reasoning. Multiple tasks were used in the experiment, and training and testing were performed on an A100 GPU using a graphics card with 32GB of memory. This experimental setup ensures the efficiency and reliability of processing large-scale data and complex models. Through this comprehensive experimental design, the performance of different models in various task areas can be comprehensively evaluated and compared. In addition, the use of high-performance hardware ensures the accuracy and repeatability of the experiment. Among them, the results in Table 1 show the evaluation results of the present invention and other methods on WebQA.

[0087] Table 1

[0088] Model Acc GPT-4 Human GPT-3.5 0.69 8 78 ChatGLM 0.6 7 74 Moss 0.43 6 64 MPT-7B-Chat 0.23 4 43 BiLL 0.37 8 63 Phoenix 0.62 5 63 ours 0.54 6 72

[0089] The results in Table 2 show the evaluation model scores of this technique and other methods under three step-by-step instructions.

[0090] Table 2

[0091] Model First instruction Second instruction Third instruction ChatGLM 76 78 78 BiLL 62 67 65 Phoenix 75 78 80 ours 80 84 82

[0092] Table 3 shows the evaluation results on the C3 task.

[0093] Table 3

[0094] Model Acc Human GPT-3.5 0.52 80 ChatGLM 0.39 65 BiLL 0.42 76 Chinese-Alpaca-LoRA 0.36 38 Phoenix 0.64 80 ERNIEBot 0.31 62 ours 0.69 82

[0095] The results in Table 4 show the performance of the evaluated models on the Ape210l and Math23k tasks.

[0096] Table 4

[0097] Model Math23k Ape210k GPT-3.5 0.56 0.37 Moss 0.31 0.11 BiLL 0.42 0.25 MPT-7B-Chat 0.11 0.07 ours 0.62 0.39

[0098] The results in Table 5 show the model performance in the idiom comprehension task.

[0099] Table 5

[0100]

[0101]

[0102] Table 6 shows the model performance in the reading comprehension task.

[0103] Table 6

[0104] Model Acc Human BiLL 0.32 60 Moss 0.52 56 Phoenix 0.63 68 ours 0.69 74

[0105] It is understood that the above model selection is based on the degree of similarity in model parameters with the Chinese reasoning task model of the present invention. The evaluation results for the main tasks are shown below. In the entertainment theme category, the GPT-3.5 model gave the answer "Celebrity A and Celebrity B held their wedding at XXX Church in the UK," while the Phoenix model gave the answer "Celebrity A and Celebrity B's British Wedding: A Love Story in Youthful Memories." The Chinese comprehension task model gave the answer "Celebrity A and Celebrity B got married," while the BiLLa model gave the answer "Celebrity A and Celebrity B held their wedding at XXX Church in the UK, concluding their lifelong event under the winess of two pastors." In the theme category, the GPT-3.5 model gave the answer "Title: Citizens of Y City, X Province, Visit the New Year's Goods Fair and Enjoy a Free Qiong Opera Feast," while the Phoenix model gave the answer "Qiong Opera Cultural Tour in Y City: Free Performances, Experience the Rich Qiong Opera Atmosphere." The Chinese comprehension task model gave the answer "Hometown of Qiong Opera - Y City. Experience Traditional Qiong Opera from X Province," while the BiLLa model gave the answer "Hometown of Qiong Opera in X Province, Y City." In the category of people's livelihood topics, the GPT-3.5 model gave the answer "The State Food and Drug Administration announced: Foods that may cause serious health damage or even death after consumption are subject to recall. Producers should initiate the recall within 24 hours after becoming aware of the food safety risk and complete the recall within 10 working days." The Phoenix model gave the answer "XX department issued the "Food Recall Management Measures". The first-level food recall must be initiated within 24 hours and completed within 10 working days." The Chinese comprehension task model gave the answer "XX department issued the "Food Recall Management Measures". The BiLLa model gave the answer "XX department announced that foods that may cause serious health damage or even death after consumption are subject to first-level recall. Food producers should initiate the recall within 24 hours after becoming aware of the food safety risk." In the social news topic category, the GPT-3.5 model answered, "A car accident resulted in 9 deaths and 24 injuries. The S City Traffic Police Bureau reported that the driver's speeding and improper operation caused the accident. Currently, 8 of the 24 injured have recovered and been discharged. Compensation is expected to exceed 10 million yuan." The Phoenix model answered, "A car, a female driver, 9 deaths and 24 injuries: The S City Traffic Police Bureau reported the cause of the accident." The Chinese comprehension task model answered, "The S City Traffic Police Bureau received a report: A car driver speeding caused 9 deaths and 24 injuries, and compensation is 10 million yuan." The BiLLa model answered, "A female driver speeding caused 9 deaths and 24 injuries at S Airport: 9 deaths and 24 injuries, follow-up: The driver is fully responsible, and compensation may exceed 10 million yuan." Through experimental comparisons of the Chinese reasoning task model with other Chinese models across four different topic categories, as well as answer length evaluation tests on the WebQA dataset, the Chinese reasoning task model surpassed existing models such as GPT-3.5, ChatGLM, and ERNIEBot in benchmark tasks such as Chinese dialogue understanding, mathematical reasoning, and idiom comprehension, demonstrating its excellent Chinese comprehension and mathematical reasoning capabilities. In addition, the model shows high stability when facing different instructions.Its performance is superior to models with similar parameters (such as Chinese-Alpaca-LoRA) in knowledge question answering and open-ended questions.

[0106] This demonstrates that this application incorporates automatic cognitive chaining instructions into an improved, post-secondary training Chinese reasoning task model. By dynamically sampling these instructions, the model is encouraged to generate reasoning chains, thereby improving answer inference and reasoning performance. This effectively reduces errors in answer generation and significantly improves the model's overall performance. Extensive comparative experiments demonstrate the superior performance of the proposed Chinese comprehension task model across various Chinese language tasks. It surpasses state-of-the-art models, including GPT-3.5 and ChatGLM, providing more accurate, comprehensive, and professional answers.

[0107] Reference Figure 6 As shown, the embodiment of the present invention also discloses a Chinese reasoning task model construction device, including:

[0108] A model construction and training module 11 is used to perform model training on the low-order adaptive model and the large language model using the Chinese dialogue dataset, so as to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training;

[0109] A model training module 12 is configured to perform secondary training on the initial Chinese comprehension task model using a Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model;

[0110] The model updating module 13 is used to perform human feedback reinforcement learning on the Chinese reasoning task model after the secondary training, and obtain the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after the secondary training, and use the relative entropy to update the model to obtain the current updated Chinese reasoning task model.

[0111] It can be seen that the present application discloses the use of a Chinese dialogue dataset to train a low-order adaptive model and a large language model respectively, so as to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training; the initial Chinese understanding task model is trained twice using a Chinese reasoning training dataset to obtain a Chinese reasoning task model after secondary training; the Chinese reasoning task model after secondary training is reinforced with human feedback, and the relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training is obtained, and the model is updated using the relative entropy to obtain the current updated Chinese reasoning task model. It can be seen that the Chinese reasoning task model constructed by the low-order adaptive model and the large language model ensures that the large language model maintains excellent task performance while combining the low-rank fine-tuning of the large language model by the low-order adaptive model, effectively optimizes parameter efficiency, reduces resource requirements, and solves the inherent limitations of traditional reinforcement learning by incorporating human feedback into the training process. The introduction of human feedback significantly enhances the training process, thereby improving the performance of the Chinese understanding task model.

[0112] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0113] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the Chinese language comprehension task model construction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0114] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0115] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0116] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0117] Among them, the operating system 221 is used to manage and control the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to calculate and process the massive data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the Chinese language comprehension task model construction method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks. In addition to including data received by the electronic device and transmitted from external devices, the data 223 can also include data collected by its own input and output interface 25.

[0118] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned method for constructing a Chinese language comprehension task model. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0120] Professionals can also further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0121] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0122] The above is a detailed introduction to the Chinese reasoning task model construction method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for constructing a Chinese reasoning task model, characterized in that: include: The low-level adaptive model and the large language model are trained using the Chinese dialogue dataset to obtain an initial Chinese understanding task model constructed by the low-level adaptive model and the large language model after initial training. Performing secondary training on the initial Chinese comprehension task model using a Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model; Performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training, and obtaining a relative entropy between a reinforced output result of the reinforced Chinese reasoning task model and an output result of the Chinese reasoning task model after secondary training, and using the relative entropy to update the model to obtain a currently updated Chinese reasoning task model; The method of using the Chinese dialogue dataset to train the low-order adaptive model and the large language model respectively to obtain an initial Chinese understanding task model constructed by the low-order adaptive model and the large language model after initial training includes: Obtain a Chinese dialogue dataset, and perform encoding and decoding on the Chinese dialogue dataset to obtain Chinese dialogue sentence vectors; The low-rank matrix of the low-order adaptive model is used to freeze the model weight parameters of the large language model to obtain a low-rank fine-tuned large language model; Using the Chinese dialogue sentence vectors to train a low-order adaptive model and a low-rank fine-tuned large language model to obtain a low-order adaptive model and a large language model after initial training; Merging the output result vectors of the low-order adaptive model and the large language model after the initial training to obtain an understanding skill matrix representing the degree of understanding of the Chinese conversation intent; Constructing an initial Chinese comprehension task model including the comprehension skill matrix based on the low-order adaptive model and the large language model after the initial training; The method of performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model includes: Obtain a Chinese reasoning training data set, and encode and decode the Chinese reasoning training data set to obtain a Chinese reasoning sentence vector; Using the Chinese reasoning sentence vectors, the thinking reasoning of the initial Chinese comprehension task model is trained to obtain a reasoning Chinese comprehension task model including a reasoning ability skill matrix; the reasoning ability skill matrix represents the degree of reasoning about the context of the Chinese sentence dialogue; The reasoning ability skill matrix in the Chinese comprehension task model is connected with the understanding skill matrix in the initial Chinese comprehension task model to generate a secondary trained Chinese reasoning task model including the understanding reasoning feature skill matrix.

2. The Chinese reasoning task model construction method according to claim 1, characterized in that: After performing secondary training on the initial Chinese comprehension task model using the Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model, the method further includes: Inputting a Chinese test question into the second-trained Chinese reasoning task model, so that the second-trained Chinese reasoning task model determines a corresponding target instance in a thought chain instance according to the question type of the Chinese test question; The target instance is used to generate a corresponding reasoning train of thought, so as to generate a Chinese test answer corresponding to the Chinese test question based on the reasoning train of thought.

3. The Chinese reasoning task model construction method according to claim 2, characterized in that: The performing human feedback reinforcement learning on the Chinese reasoning task model after secondary training includes: The Chinese reasoning task model after secondary training that outputs the Chinese test answer is subjected to human feedback reinforcement learning, the answer matrix constructed by the Chinese test answer is scored, and the Chinese test answer with the highest score is selected from all scores and output as the final output result.

4. The Chinese reasoning task model construction method according to claim 2, characterized in that: Before inputting the Chinese test question into the second-trained Chinese reasoning task model so that the second-trained Chinese reasoning task model determines the corresponding target instance in the thinking chain instance according to the question type of the Chinese test question, the method further includes: Using a preset clustering algorithm and dividing the Chinese question training samples into different numbers of Chinese question training clusters according to question types; Sort the Chinese question training samples in each Chinese question training cluster in ascending order according to their distance from the cluster center to create a Chinese question matrix table; A sample matrix table including a question matrix and an answer matrix is ​​generated based on answer training samples and question training samples in the Chinese question training samples and normalized.

5. The method for constructing a Chinese reasoning task model according to any one of claims 1 to 4, characterized in that: The method of using the relative entropy to perform model updating to obtain a currently updated Chinese reasoning task model includes: The relative entropy is set to a negative value, and the relative entropy is added to the reward signal of the reward model to update the reward model, and a loop operation of human feedback reinforcement learning is performed on the Chinese reasoning task model after secondary training using the updated reward model until the number of cycles reaches a preset threshold, and the updated Chinese reasoning task model is output.

6. A Chinese reasoning task model construction device, characterized in that: include: A model building and training module is used to train the low-level adaptive model and the large language model using the Chinese dialogue dataset to obtain an initial Chinese understanding task model constructed by the low-level adaptive model and the large language model after initial training; A model training module is used to perform secondary training on the initial Chinese comprehension task model using a Chinese reasoning training dataset to obtain a secondary trained Chinese reasoning task model; a model updating module, configured to perform human feedback reinforcement learning on the Chinese reasoning task model after secondary training, obtain a relative entropy between the reinforced output result of the reinforced Chinese reasoning task model and the output result of the Chinese reasoning task model after secondary training, and use the relative entropy to perform model updating to obtain a currently updated Chinese reasoning task model; The model construction training module is specifically used to obtain a Chinese dialogue dataset, encode and decode the Chinese dialogue dataset to obtain a Chinese dialogue sentence vector; use the low-rank matrix of the low-order adaptive model to freeze the model weight parameters of the large language model to obtain a low-rank fine-tuned large language model; use the Chinese dialogue sentence vector to train the low-order adaptive model and the low-rank fine-tuned large language model to obtain the low-order adaptive model and the large language model after initial training; merge the output result vectors of the low-order adaptive model and the large language model after initial training to obtain an understanding skill matrix that represents the degree of understanding of the Chinese dialogue intention; construct an initial Chinese understanding task model containing the understanding skill matrix based on the low-order adaptive model and the large language model after initial training; The model training module is specifically used to obtain a Chinese reasoning training data set, encode and decode the Chinese reasoning training data set to obtain a Chinese reasoning sentence vector; use the Chinese reasoning sentence vector to train the thinking reasoning of the initial Chinese comprehension task model to obtain a reasoning Chinese comprehension task model including a reasoning ability skill matrix; The reasoning ability skill matrix represents the degree of reasoning about the context of Chinese sentences; The reasoning ability skill matrix in the Chinese comprehension task model is connected with the understanding skill matrix in the initial Chinese comprehension task model to generate a secondary trained Chinese reasoning task model including the understanding reasoning feature skill matrix.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the Chinese reasoning task model construction method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the Chinese reasoning task model construction method according to any one of claims 1 to 5 are implemented.

Images