Agricultural field large language model training method and device and medium

Abstract

The invention provides an agricultural field large language model training method and device and a medium, and the method comprises the steps: collecting agricultural field data book information, and carrying out the preprocessing of the information, and obtaining a first Chinese data set and a first minority language data set, collecting the first Chinese data set and the first minority language data set, generating a first mixed language data set, processing the first mixed language data set by adopting a word segmentation library, performing secondary processing by adopting a preset special character processing method, constructing a word segmentation list, and constructing a WordPiece word piece model to process and divide the word segmentation list; converting each word in the word segmentation list into a word vector by adopting a word embedding algorithm; the method comprises the following steps of: constructing a coding component and a decoder component which are based on a Transform architecture and adopt a self-attention mechanism, generating an agricultural field large language model, and training the agricultural field large language model by taking a word vector as an input to obtain a trained agricultural field large language model. The problem of supporting a large language model of Chinese and minority languages is solved.

Description

Technical Field

[0001] This document relates to the field of large prediction model technology, and in particular to a large prediction model training method, equipment and medium in the agricultural field. Background Art

[0002] In terms of general large language models, there are many general large language models publicly released internationally, such as ChatGPT, Wenxin Yiyan, iFlytek Spark, Tongyi Qianwen, ChatGLM, BERT, etc. ChatGPT supports English, Chinese, Spanish, French, German, Russian, etc.; Wenxin Yiyan mainly supports Chinese, English, Japanese, and Korean; iFlytek Spark mainly supports Chinese and English; Tongyi Qianwen mainly supports Chinese and English; ChatGLM mainly supports Chinese and English; BERT mainly supports English, Chinese, French, and Spanish, etc. The above general large language models have certain defects: ① The supported languages ​​are limited, for example, there is no lightweight large language model that can support minority languages. ② It is impossible to achieve lightweighting efficiently. The general large language model requires massive amounts of data from various categories during the training process. The trained models are often huge in size. In subsequent applications, they can only be lightweighted and compressed through cost-reduction technology. This method cannot fundamentally solve the problem of model lightweighting; ③ There is a certain delay effect on policy knowledge learning; ④ The model is too large and has high requirements for hardware resources. It cannot currently run normally on a non-GPU personal PC.

[0003] In terms of the application of general large language models in professional fields, the professional field models currently publicly released internationally include models in the fields of medical health, legal services, finance, etc. These models are based on existing general large language models such as ChatGPT and ChatGLM as the base models, pre-trained professional knowledge bases and fine-tuned. The disadvantage of these models is that they use general models to complete the underlying reasoning, that is, the language reasoning mechanism is single.

[0004] The present invention proposes a large prediction model training method in the agricultural field, which supports a bilingual reasoning mechanism for Chinese and minority languages; proposes a professional knowledge base corpus collection, organization and vocabulary construction that supports Chinese and minority languages; and obtains multiple basic models by training agricultural branch fields. The basic models are customized and combined to solve the problem of lightweight models. Summary of the invention

[0005] The present invention provides a large prediction model training method, equipment and medium in the agricultural field, aiming to solve the above problems.

[0006] The embodiment of the present invention provides a large prediction model training method in the agricultural field, comprising:

[0007] An embodiment of the present invention provides an electronic device, including:

[0008] processor; and,

[0009] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the above-mentioned large language model training method in the agricultural field.

[0010] An embodiment of the present invention provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the above-mentioned large language model training method in the agricultural field.

[0011] The embodiment of the present invention adopts a bilingual reasoning mechanism that supports Chinese and minority languages; proposes the collection, organization and vocabulary construction of professional knowledge base corpora that support Chinese and minority languages; obtains multiple basic models by training agricultural branch fields, and the basic models are customized and combined to solve the problem of lightweight models. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0013] Figure 1 For the embodiments of the present invention;

[0014] Figure 2 Shown is an overall working operation diagram of an embodiment of the present invention;

[0015] Figure 3 A schematic diagram of a method and class for constructing a large prediction model in the agricultural field according to an embodiment of the present invention;

[0016] Figure 4 A schematic diagram of a training model according to an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of a model architecture for an embodiment of the present invention;

[0018] Figure 6 The first part of the specific implementation code of the application of the lightweight agricultural question-answering robot based on the custom GPT model in the embodiment of the present invention;

[0019] Figure 7 The second part of the specific implementation code of the application of the lightweight agricultural question-answering robot based on the custom GPT model in the embodiment of the present invention;

[0020] Figure 8 The third part is the specific implementation code of the application of the lightweight agricultural question-answering robot based on the custom GPT model in the embodiment of the present invention;

[0021] Fig. 9 This is the fourth part of the specific implementation code for the application of a lightweight agricultural question-answering robot based on a custom GPT model in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0023] Glossary:

[0024] 1. Mask: Mask means mask, which covers certain values ​​so that they have no effect when the parameters are updated. There are two types of masks involved in the Transformer model, namely Padding Mask and Sequence Mask. Padding Mask is required in all scaled dot-product attention, while Sequence Mask is only used in the Self-Attention of the decoder.

[0025] 2. Padding Mask

[0026] That is, fill the shorter sequence with 0 (but if the input sequence is too long, it is truncated and the excess is discarded directly). Because these filled positions are actually meaningless, the Attention mechanism should not focus on these positions, so some processing is needed.

[0027] 3. Sequence Mask

[0028] Sequence Mask is used to prevent the decoder from seeing future information. That is, for a sequence, the decoded output should only depend on the output before the moment, and not on the output after. Because we need to find a way to hide the future information.

[0029] Specific approach: Generate an upper triangular matrix with all upper triangular values ​​set to 0. Apply this matrix to each sequence to achieve our goal. For the Decoder's Self-Attention, the scaled dot-product attention used in it requires both Padding Mask and Sequence Mask. The specific implementation is to add the two Masks. In other cases, only Padding Mask is needed.

[0030] 4. Principle of forward propagation

[0031] In the forward propagation of Transformer, the following steps are mainly included:

[0032] Position encoding: Adding position encoding to the embedding vector of the input sequence enables the model to distinguish information at different positions.

[0033] Self-attention layer: Through the above-mentioned Query-Key-Value calculation and attention score calculation, the attention distribution of each position is determined, and then the Value matrix is ​​weighted and summed according to this distribution to obtain a new representation.

[0034] Multi-head attention: The above self-attention process will be executed multiple times in parallel on different subspaces (i.e., multi-head), and finally the results will be concatenated and linearly transformed to increase the expressive power of the model.

[0035] Feed Forward Networks (FFN): After the self-attention layer, the vector at each position will pass through two linear transformations (with a nonlinear activation function such as ReLU in between) of FFN to further extract features.

[0036] 5. Back propagation principle

[0037] During the back-propagation process, the goal of the Transformer is to minimize the loss function between the predicted output and the true label. The specific steps include:

[0038] Calculate the loss: First calculate the loss between the model output and the true label (such as cross entropy loss).

[0039] Gradient calculation: Using the chain rule, we backpropagate the error from the loss function and calculate the gradient of each layer’s weights and biases. In the self-attention layer and FFN, a large number of matrix multiplications and derivative calculations of nonlinear functions are involved.

[0040] Weight update: Update the model's weights using the calculated gradients and an optimization algorithm (such as Adam, SGD). Typically, the update rule is the current weight minus the learning rate multiplied by the gradient.

[0041] Iterative process: This process is repeated multiple times on the entire dataset (or small batches of data) until the model converges or reaches a predetermined number of training rounds.

[0042] 6. Encoding and decoding principles

[0043] Encoding: In the GPT model, "encoding" is mainly reflected in the word embedding and position encoding stages. Word embedding maps each word in the vocabulary to a high-dimensional vector space, while position encoding assigns a unique vector to each position in the sequence. Combining the two, the model can understand each word and its position information in the sentence.

[0044] Decoding: The decoding process is essentially the process of the model predicting the next word or words based on the input sequence. In GPT, this is achieved by gradually extracting the contextual information of the sequence through the self-attention mechanism of the Transformer and the feedforward network layer by layer, and converting it into a probability distribution for the vocabulary through the final linear layer. The generation process is carried out by selecting the word with the highest probability or sampling method, and then adding the newly generated word to the sequence, and repeating the process until the preset maximum number of new words is reached.

[0045] 7. Principle of attention mechanism

[0046] The self-attention mechanism allows the model to dynamically focus on different parts of the input sequence based on different attention weights when processing it. This is divided into three steps:

[0047] Query, Key, and Value calculations: The input sequence is transformed into query (Q), key (K), and value (V) matrices through linear transformation;

[0048] Attention weight calculation: For each query, calculate its dot product with all keys and convert it into a probability distribution, i.e., attention weight, through the softmax function. In this step, a causal mask is applied to ensure that only past words can be seen during generation, which conforms to the time order of natural language.

[0049] Weighted summation: The value matrix is ​​weighted and summed using the attention weights to obtain a new representation, which emphasizes the part of the input sequence that is relevant to the current position.

[0050] Method Embodiment

[0051] According to an embodiment of the present invention, a method for training a large prediction model in the agricultural field is provided. Figure 1 This is a flowchart of the agricultural field large prediction model training method according to an embodiment of the present invention. Figure 1 As shown, the agricultural field large prediction model training method of the embodiment of the present invention specifically includes:

[0052] S1, collecting agricultural data and book information, and pre-processing to obtain a first Chinese data set and a first minority language data set, and merging the first Chinese data set and the first minority language data set to generate a first mixed language data set; S1 specifically includes:

[0053] S11, collecting data and book information in the agricultural field, classifying them according to the fields involved, and then performing text conversion to obtain a first data set;

[0054] In the embodiment of the present invention, the book service purchased by University A mainly downloads pdf book information, which is divided into twelve categories (agricultural development history, crop planting, animal husbandry, forestry, aquaculture, agricultural product processing, agricultural machinery, agricultural economy, agricultural environment, agricultural technology, agricultural policy, and agricultural and rural development). Image recognition is performed on the pdf. Each page of the pdf book information file is a text presented in the form of an image. Each page is further identified and converted, and the converted text is stored in a .txt document corresponding to the book name.

[0055] S12, cleaning the first data set and removing garbled characters to obtain a second data set;

[0056] S13, converting the second data set into simplified Chinese characters to obtain a third data set;

[0057] Some books are presented in traditional and simplified Chinese, so it is necessary to convert traditional Chinese to simplified Chinese. Our Chinese training set is uniformly presented in simplified Chinese, and we use Python's opencc library for batch conversion.

[0058] Merging the third data set based on the knowledge graph structure to obtain a first Chinese data set;

[0059] The coherence of knowledge points and the sufficiency of sentences for knowledge points of the same category largely determine the context coherence of the model-generated text content and the accuracy of text extraction. The data merging method of the embodiment of the present invention uses the knowledge graph structure as the basis to integrate the book materials and article materials published by the knowledge graph structure elements. For example, the contents of the knowledge point of the concept of "agriculture" in multiple books are merged to meet the coherence of the knowledge extraction context and the distance proximity characteristics of the relevant context weights. Paragraphs are spliced ​​without blank lines, and the knowledge points in the two books on agricultural informatization and new rural agricultural informatization knowledge questions and answers: "What is agricultural informatization?" Part of the content and "What does agricultural informatization mean?" Part of the content is merged; the sentence knowledge of the "What is agricultural information technology?" part is merged, and the beginning part of the merged content is used <cllasify-start>Tag mark, the end position is marked with <cllasify-end>Label marking, merge the numbers of the twelve book categories separately, and merge each small knowledge point with <cllasify-start>The tag starts with <cllasify-end>The tag ends. The purpose of this is to mark the subsequent text context coherence detection and output fuse mechanism. Experiments show that the more sufficient the context length is, the richer the sentence features recognized by the model, the more accurate the knowledge extraction and generation prediction is, and the more reasonable the sentence expression is.

[0060] S14, translating the first Chinese data set into a minority language and performing grammar correction to obtain a first minority language data set;

[0061] Training a large language model for minority languages ​​requires a large amount of Uyghur corpus data, but there are very few agricultural books and high-quality articles in minority languages. Therefore, a translation crawler program for the minority language translation interface is built to translate the Chinese texts that have been sorted, classified and merged above, and store them in the ug minority language classification directory. The grammatical errors in the text translated by the minority language translation interface are corrected.

[0062] S15: Merge the first Chinese data set and the first minority language data set according to classification to generate a first mixed language data set.

[0063] S2, after processing the first mixed language data set using a word segmentation library, performing secondary processing using a preset special character processing method, and constructing a word segmentation table, and constructing a WordPiece word piece model to process and divide the word segmentation table;

[0064] The cleaned and organized data is segmented and a vocabulary (thesaurus) is constructed to provide a basis for subsequent model training.

[0065] Chinese and minority language corpus word segmentation. Chinese corpus word segmentation mainly uses the jieba library for word segmentation. Because jieba has a very strong vocabulary library for Chinese word segmentation and does a good job in the field of Chinese word segmentation, but it cannot handle special marks in a special way, so this project uses jieba word segmentation and then performs secondary processing to retain some special marks and characters. The special marks customized by the embodiment of the present invention include the start mark of the section, the start mark of the paragraph, the space mark, the line break mark, the symbol mark, and the like.

[0066] For the first special tag in the above sub-item, by giving the function a document address name, this special tag function reads each tag in the above text document line by line and returns a list of special tags.

[0067] Define the segment_text_with_custom_token function, which defines two parameters. The first parameter seg_list is the word segmentation table after Jieba segmentation, and the second parameter cn_word_file is the file address of the special tag document. The special tag address is obtained by calling the read_special_tokens function and passing in the text document address. Such as: cn_word_list = read_special_tokens(cn_word_file); and through a loop through each special tag, each special tag is segmented by jieba in turn, and each symbol after the single special tag is split is further traversed; each split symbol is searched in the passed seg_list dictionary and the retrieved symbols used in the dictionary are deleted; the seg_list is searched and deleted, and the vocabulary obtained after the incomplete words after the initial split is used to traverse the complete special tag list of cn_word_list once and add it to the cleaned seg_list vocabulary, and the size of the vocabulary is calculated by len(seg_list)+500, and the special tag is used as the key. 500 is added because some special tags are deleted above. At present, the actual id after deletion is larger than the size of the dictionary. The tag expansion of 500 words in this project is sufficient to ensure the uniqueness of the key. The current vocabulary is stored in the seg_list dictionary as a value, and a new dictionary is returned. This ensures that the newly added vocabulary id is unique; the custom dictionary adjusts the list of segmented words to ensure that the custom vocabulary is treated as a separate term and assigns it an index.

[0068] S3, using word embedding algorithm to convert each word in the word segmentation table into a word vector;

[0069] The embodiment of the present invention uses the tokenizers library to train a WordPiece word piece model for text processing tasks, and processes and divides each word in the word segmentation table (derived from the JSON vocabulary table obtained by the above processing), and finally saves the processed data. It is divided into several key steps: loading the vocabulary table, dividing the data set, training the WordPiece model, and simulating and saving the word segmentation data. Each part is described in detail below.

[0070] Function and characteristics: By reading the JSON file, all words in the vocabulary are extracted as the basic data for training the WordPiece model. This step ensures that the model training is based on known domain vocabulary and improves the accuracy of subsequent models in text processing in specific fields.

[0071] Logic: Use Python's json.load() function to read the JSON file content, and then extract all the keys (i.e. vocabulary items) in the key-value pairs to form a list.

[0072] Dataset Partitioning

[0073] Function and characteristics: Split the loaded vocabulary dataset into training set and test set according to a given ratio (90% training, 10% test by default) to support effective training and performance verification of the model.

[0074] Logic: Use simple array slicing operations to partition the data set to ensure that the training and test data are balanced and representative.

[0075] Training WordPiece word piece model

[0076] Function and Features: Use Hugging Face's tokenizers library to train a WordPiece model. WordPiece is a common text encoding method that effectively handles vocabulary coverage issues by splitting rare or unknown words into known subword units.

[0077] Logic: First, define a Tokenizer instance and configure the WordPiece model parameters, including special symbols, maximum input character length, etc. Then, train the model with training data through iterators, and specify training parameters such as vocabulary size and minimum word frequency. Finally, save the trained model to the specified directory.

[0078] Simulate saving the segmented data to a file

[0079] Functional characteristics: Although this part does not actually perform word segmentation operations (because the trained tokenizer is not used to process the data), its design intention is to show how to save the processed data for further analysis or verification. This step is a theoretical follow-up operation prompt.

[0080] Logic: Simply write the raw data into two files line by line, representing the training set and the test set respectively. In practical applications, this step should be performed after word segmentation, and the text should be processed with the trained tokenizer before saving.

[0081] By customizing the WordPiece model, you can optimize vocabulary coverage and adapt to professional terms and expressions in specific fields, thereby improving the understanding and accuracy of subsequent models (such as question-answering, classification, or generation models) when processing agricultural-related texts. The reasonable division of the data set ensures the effectiveness of model training and the feasibility of generalization ability evaluation. Although the last step only illustrates data preservation, the complete process reflects the basic framework from data preparation to model training, providing a solid foundation for the development of NLP applications in specific fields.

[0082] S4. Construct an encoding component and a decoder component based on the Transformer architecture with a self-attention mechanism to generate a large language model in the agricultural field;

[0083] Transformer can be divided into two parts: encoding component and decoding component:

[0084] The encoding component is composed of multiple layers of encoders (a 16-layer encoder is used in the invention, and you can try other layers in actual use). The decoding component is also composed of decoders (Decoders) with the same number of layers.

[0085] Each encoder consists of two sublayers: the Self-Attention layer and the Position-wise Feed Forward Network (FFN); the structure of each encoder is the same, but they use different weight parameters (the architecture of the 16 encoders is the same, but the parameters are different), and the output of the Self-Attention layer flows into the feedforward network. The input of the encoder flows into the Self-Attention layer first. It allows the encoder to use the information of other words in the input sentence when encoding a specific word (it can be understood as: when we predict a word, we not only pay attention to the current word, but also pay attention to the information of other words).

[0086] The decoder also has these two layers in the encoder, but there is an attention layer between them (i.e. Encoder-Decoder Attention) to allow the decoder to focus on the relevant parts of the input sentence.

[0087] The encoder includes: self-attention layer and feedforward network FFN (Position-wise Feed Forward Network);

[0088] The decoder includes: self-attention layer, Encoder-Decoder Attention and feed-forward network FFN (Position-wise Feed Forward Network);

[0089] Each word is converted into a word vector using the word embedding algorithm. The dimension of the word embedding vector is 512, and each word is embedded in a vector of size 512. These vectors will be represented by these simple boxes.

[0090] Word embedding only happens in the bottom-level encoder. All encoders receive a list of vectors of size 512:

[0091] The bottom encoder receives the word embedding vector, and the other encoders receive the output of the previous encoder.

[0092] This list size is a hyperparameter that we can set - basically it's the length of the longest sentence in our training dataset.

[0093] Word embedding post-encoding

[0094] After the embedding operation is completed for the input sequence, each word will flow through the two layers of the encoder. The following figure shows word embedding and encoding:

[0095] The encoder receives a vector as input. The encoder first passes these vectors to the Self-Attention layer, then to the feed-forward network, and finally passes the output to the next encoder.

[0096] Self-Attention:

[0097] As the model processes each word (each position in the input sequence), the Self-Attention mechanism enables the model to focus not only on the word at the current position, but also on words at other positions in the sentence, so that it can better encode the word. Transformer uses the Self-Attention mechanism to incorporate the understanding of other words into the current word.

[0098] Multi-head attention mechanism architecture

[0099] The self-attention layer is further improved by adding a multi-head attention mechanism. The specific steps are as follows:

[0100] First, the query, key, and value are mapped through different linear transformations; then, different attentions are concatenated; finally, another linear transformation is performed. Each set of attention is used to map the input to a different sub-representation space, which allows the model to focus on different positions in different sub-representation spaces.

[0101] The position feedforward network is a fully connected feedforward network. The words at each position pass through this identical feedforward neural network separately. It consists of two linear transformations, namely two fully connected layers. The activation function of the first fully connected layer is the ReLU activation function. In each encoder and decoder, although the fully connected feedforward network structure is the same, no parameters are shared. The input and output dimensions of the entire feedforward network are, the output dimension of the first fully connected layer and the input dimension of the second fully connected layer are, residual connection and layer normalization. There is a detail that needs to be noted in the encoder structure: each sublayer of each encoder (Self-Attention layer and FFN layer) has a residual connection, and then a layer normalization operation is performed.

[0102] There is one thing missing from the models described so far: a way to represent the order of words in a sequence. To solve this problem, the model adds a vector to each input word embedding vector. These vectors follow a specific pattern learned by the model and help the model determine the position of each word, or the distance between different words in the sequence. But the advantage of this approach is that it can be extended to unknown sequence lengths.

[0103] Decoder: Through the above steps, the input of the first encoder is a sequence, and the output of the last encoder is a set of attention vectors Key and Value. These vectors will be used in each decoder Encoder-Decoder Attention layer, which helps the decoder focus on the appropriate position of the input sequence. After completing the encoding phase, we start the decoding phase. Each time step in the decoding phase outputs an element. The working principle of the Encoder-Decoder Attention layer is similar to the multi-head self-attention mechanism. The difference is that the Encoder-Decoder Attention layer uses the output of the previous layer to construct the Query matrix, while the Key and Value matrices come from the output of the encoder stack.

[0104] Final Linear and Softmax Layers: The output of the decoder stack is a float vector. This is done by a linear layer followed by a softmax layer. The linear layer is a simple fully connected neural network that maps the output vector of the decoder stack to a longer vector called the logits vector.

[0105] Softmax layer: Now assume that the model has 240,000 Chinese words (the output vocabulary of the model). Therefore, the logits vector has 240,000 numbers, each representing the score of a word. Then, the Softmax layer converts these scores into probabilities (converts all scores to positive numbers and adds up to 1). Finally, the word corresponding to the highest probability is selected as the output of this time step.

[0106] In order to improve the performance of the model, the following regularization operations are used during training. Dropout operations are used on the output of each sub-layer of the encoder and decoder before residual connection and layer normalization. Dropout operations are performed after the word embedding vector and the position encoding vector are added.

[0107] The matrix multiplication in the model is not only used for basic linear transformation, but also a key operation for calculating attention scores and information integration. Forward propagation and back propagation are the basis of model learning and optimization. The former is used for prediction, and the latter is used to adjust model parameters to reduce prediction errors.

[0108] Generative large language models usually adopt Transformer-based architectures, such as BERT, GPT, ChatGLM, iFlytek Spark, Wenxing Yiyan, Tongyi Qianwen, etc. These models are composed of multiple self-attention mechanisms and feedforward neural networks, which can effectively capture long-distance dependencies in texts. Large language models are all implemented based on the Transformer architecture. The differences are that the corpus implemented by each large language model, the encoder and decoder constructed by Transformer, the attention mechanism, the matrix dimension of matrix multiplication to calculate cosine similarity, the model function, size, the language types involved in training, and the configuration parameters are different.

[0109] Through the above operations, a model based on the Transformer architecture is implemented, which is mainly used for text generation tasks. The following is a detailed analysis of its main functions, technical points and potential innovations:

[0110] Key features include:

[0111] 1. Text generation: By giving some initial text (seed text), the model is able to generate continuous text sequences, which is very useful for areas such as automatic text creation and dialogue systems.

[0112] 2. Self-attention mechanism: The core of the model is the SelfAttention module, which implements the multi-head self-attention mechanism, which allows the model to consider the contextual information of the entire input sequence when generating each word.

[0113] 3. Position encoding: This is implemented using the SinusoidPE class, which appends a unique vector to each position in the input sequence, allowing the model to distinguish words at different positions. This is the key to the Transformer model processing sequential data.

[0114] 4. Residual connection and layer normalization: Each Decoder Block contains LayerNorm and residual connection, which helps to train deep networks and accelerate convergence.

[0115] 5. Feed Forward Networks (FFN): After the self-attention layer, FFN further extracts features to enhance the expressiveness of the model.

[0116] 6. Embedding layer and output layer: The embedding layer is used to map the words in the vocabulary to a high-dimensional vector space, and finally a linear layer is used to convert the output of the model back to the probability distribution of the vocabulary size.

[0117] Technical points include:

[0118] 1. Multi-head self-attention: By splitting the attention mechanism into multiple "heads", the model can focus on different features at different positions at the same time, increasing the model's parallel computing and learning capabilities.

[0119] 2. Position encoding: Generated using sine / cosine functions, which is related to the absolute position, allowing the model to capture sequence information without explicitly using a recurrent structure.

[0120] 3. Masking: Using masks in self-attention calculations ensures that only past sequence information is used to predict the current word, which meets the causal requirements of the language model.

[0121] 4. Dropout: Apply dropout in self-attention and feed-forward networks to reduce overfitting.

[0122] 5. Parameter initialization: Using specific initialization methods, such as normal distribution initialization, helps to stabilize the training of the model.

[0123] Innovations include:

[0124] 1. More efficient attention mechanism: The current model uses the standard self-attention mechanism. As the sequence length increases, its computational complexity will grow rapidly. In order to improve efficiency, you can consider using Locality Sensitive Hashing Attention (LSH Attention). LSH Attention uses a hash function to divide similar key-value pairs into the same bucket, so that only the attention within the bucket is calculated, which greatly reduces the amount of calculation, especially on long sequences. This method significantly reduces the computational cost of self-attention while maintaining high performance, and is particularly suitable for large-scale language models.

[0125] 2. Adaptive Positional Encoding: Standard Sinusoidal Positional Encoding is static and cannot adapt to changes in sequence length and complexity. Adaptive positional encoding techniques, such as Adaptive Embeddings or Relative Positional Encoding, can dynamically adjust the positional encoding according to the actual length and context of the input sequence, enhancing the model's understanding of the sequence structure. Relative Positional Encoding takes into account the relative position between words, and is particularly suitable for language tasks that require understanding long-distance dependencies, improving the model's generalization ability and learning efficiency.

[0126] 3. Model structure optimization: variable-length Transformer layer: Allowing different layers of Transformer to have different sequence processing lengths helps the model capture different ranges of contextual information at different levels. For example, the bottom layer can focus on local details, while the upper layer focuses on global semantics.

[0127] 4. Improved residual connections: Explore technologies such as Residual Scaling or Multi-Branch Residuals. The former stabilizes training by introducing a scaling factor on the residual connection, while the latter increases the diversity of the model by introducing a multi-branch structure. Both can effectively improve the training stability and expressiveness of the model.

[0128] 5. Lightweight design:

[0129] 5.1 Model compression: Reduce the size of the model through quantization, pruning, knowledge distillation and other technologies. Quantization converts model parameters from floating point numbers to low-precision integers, pruning removes weights that contribute less to the model output, and knowledge distillation allows a large model (teacher model) to guide the learning of a small model (student model) to transfer knowledge, maintain performance and reduce model size.

[0130] 5.2 Lightweight architecture: Develop or adopt Transformer variants designed specifically for resource-constrained environments. The model significantly reduces the computational and memory requirements of the model while ensuring certain performance by reducing the number of layers, reducing the hidden layer dimensions, optimizing the attention mechanism, etc.

[0131] S5. Using the word vector as input, the agricultural field large language model is trained to obtain a trained agricultural field large language model.

[0132] Data loading and batch processing

[0133] Memory mapped files (np.memmap): By using numpy.memmap, large files (such as text datasets) can be virtually loaded into memory instead of having to load them all at once, which is a memory-efficient approach for large datasets. Here, training data (train.dat) and test data (test.dat) are loaded separately.

[0134] Batch data processing: The get_batch function randomly selects data fragments as batches to ensure that the model receives different parts of data at each iteration. This is crucial for generalization during training. It returns two tensors, the input sequence x and the target sequence y, where the target sequence is the next token of the input sequence, which is used to train the language model.

[0135] Training loop

[0136] Optimizer selection: The AdamW optimizer is used, which is a variant of the Adam optimizer that improves the generalization ability of the model by separating weight decay. The learning rate is set to 1e-3, which is a common initialization choice suitable for many language model training tasks.

[0137] Training steps: In each iteration, the code performs the standard process of zero gradient clearing, forward propagation, loss calculation, back propagation, and gradient update. The current loss value is printed every 100 iterations to facilitate monitoring of the training process.

[0138] Hardware acceleration: Use the device variable to determine whether to use the GPU (if available). Use .to(device) to transfer data and models to the specified device for acceleration.

[0139] Model saving and loading

[0140] Model persistence: After training, use torch.save(model.state_dict(),

[0141] "gpt_model.pth") to save the model parameters for subsequent use.

[0142] Model loading and reasoning: Although the code does not directly show the model loading process, it provides annotations for loading model parameters. The model can be restored by loading the state dictionary into the model instance. In addition, it shows how to use the loaded model and SentencePiece word segmenter for text generation, reflecting the practical application value of the model.

[0143] like Figure 2 FIG. 1 is an overall working diagram of an embodiment of the present invention. Figure 3 A schematic diagram of a method and class for constructing a large prediction model in the agricultural field according to an embodiment of the present invention. Figure 4 A schematic diagram of a training model according to an embodiment of the present invention; Figure 5 Schematic diagram of a model architecture according to an embodiment of the present invention.

[0144] The embodiments of the present invention are described in more detail with specific codes, such as Figure 6-Figure 9 The application process of a lightweight agricultural question-answering robot based on a custom GPT model is implemented:

[0145] Loading model: First, load the pre-trained GPT model weights from the specified path (such as gpt_model.pth) through the load_model function. The load_models function loops through the imported batch model address to load multiple models, initializes the model configuration using GPTConfig, and sets the model to evaluation mode (.eval()) to prepare for text generation.

[0146] Loading tokenizer: Use the load_tokenizer function to load the tokenizer model (such as bird_shooter.model) to provide the necessary tools for text encoding and decoding.

[0147] Build a chat interface: Use the gradio library to create a user interaction interface, define an input box to accept user questions or prompts (up to 7 lines), and an output box to display the responses generated by the model. The interface title is "Lightweight Professional Agricultural Question and Answer Robot" and is accompanied by explanatory text to guide user input.

[0148] Processing user input: After the user enters text on the interface, it is processed by the chatbot function. This function calls generate_text internally to generate a reply.

[0149] Text generation and decoding:

[0150] In the generate_text function, the user's input text is first encoded into a form that the model can understand using a tokenizer, and then a new text sequence is generated through the GPT model. The generated sequence is decoded back into natural language text.

[0151] Model loading and preparation:

[0152] First, load the pre-trained GPT model through the load_model function. Here, the model configuration is initialized by the GPTConfig class, and then torch.load is used to load the model weights to the CPU, and the model is set to evaluation mode (.eval()) in preparation for inference.

[0153] Input processing:

[0154] The text input by the user is encoded by the tokenizer into a tensor form that the model can understand through the sp.encode method. This process usually involves vocabulary mapping, that is, converting words into indexes.

[0155] Text Generation:

[0156] Generate new text using the .generate method of the GPT model. Here, the input context (i.e. the encoded input text) and the maximum number of new tokens generated (max_new_tokens = 150) are specified, which means that the model will generate a text sequence of up to 150 new words (or tokens) based on the given input.

[0157] Decoding the generated text: The generated sequence (in tensor form) is decoded back to natural language text through sp.decode. The decoding process is the opposite of encoding, converting the index sequence generated by the model back to the corresponding word sequence to form a coherent text.

[0158] Context coherence detection and fuse mechanism:

[0159] The generate_text_with_circuit_breaker function implements a circuit breaker mechanism to remove paragraphs in the generated text that are highly similar to the input text to ensure the freshness and diversity of the output. It calculates the cosine similarity between paragraphs and only retains the paragraph if the similarity is below the set threshold and the paragraph ends with a specific character (such as a period, exclamation mark, etc.). This approach avoids the output of duplicate information and improves the quality of the response.

[0160] Calculate cosine similarity: First, define the calculate_similarity function, and use TfidfVectorizer and cosine_similarity to calculate the cosine similarity of two texts. This step converts the text into TF-IDF vector representation, and then quantifies the semantic similarity between them through cosine similarity. Lower similarity means that the content of the two texts is quite different.

[0161] Circuit breaker implementation: The generate_text_with_circuit_breaker function implements the core logic of the circuit breaker mechanism. It receives the input text (input_text) and the original generated reply text (output_text), as well as an optional similarity threshold (default is 0.0). The function first divides the reply text into multiple paragraphs, and then calculates the similarity with the input text paragraph by paragraph. For each paragraph, if its similarity with the input text is lower than the set threshold, and the paragraph ends with a specific end character (such as a period, exclamation mark, etc.), the paragraph is considered novel and complete, so it will be retained. Finally, the retained paragraphs are reassembled into the final output text, which removes redundant information that is highly similar to the input, ensuring the independence and innovation of the output content.

[0162] Text segmentation and formatting: The split_and_output_text function is responsible for processing the generated text according to specific rules (such as \t\t delimiters or sentence segmentation) to ensure that the output format meets the requirements, such as limiting the number of sentences output.

[0163] Result feedback: After the above processing, the final generated reply text is displayed to the user through the output box of the Gradio interface, completing an interactive process. The program implements the complete process from model loading, user interaction interface construction, user input processing, model reasoning, text processing and optimization, to the final result feedback. In particular, the diversity and coherence of the generated text are enhanced through the fuse mechanism, which improves the user experience.

[0164] This model can be deployed locally on your own 4-core 8G personal PC, or on a cloud platform server. You only need to download the benchmark model file locally and run it through the python file of the above code to successfully deploy it, that is: python chatboot.py

[0165] The library packages required for this project are as follows:

[0166] torch - PyTorch 1.12 library for building, training, and reasoning deep learning models. Please make sure to install a version of PyTorch compatible with Python 3.8.

[0167] gradio - A library for quickly building interactive machine learning and deep learning web applications. The latest version can be installed via pip install gradio.

[0168] tokenizers - A library for text processing and creating custom tokenizers, especially useful for large text datasets. Install with pip install tokenizers.

[0169] NumPy - provides high-performance multidimensional array objects and tools, and is the basis of most scientific computing libraries. Usually pre-installed, otherwise install it using pip install numpy.

[0170] sentencepiece - A library for text processing, in particular for creating models of subword units. Install via pip install sentencepiece.

[0171] sklearn (scikit-learn) - machine learning library, used in this code for text feature extraction and cosine similarity calculation. Installation command: pip install -U scikit-learn.

[0172] jieba - Chinese word segmentation tool, suitable for Chinese text processing. Install via pip install jieba.

[0173] regex, or re for short, is part of the Python standard library for regular expression processing and usually does not need to be installed separately.

[0174] torchinfo-Used to print detailed structural information of PyTorch models. Installation command: pip installtorchinfo.

[0175] json- is also part of the Python standard library and is used to process JSON format data. It does not need to be installed separately.

[0176] By adopting the embodiments of the present invention, the following beneficial effects are achieved:

[0177] 1. The training cost is reduced from enterprise-level cost to civilian-level cost. It only takes one to three months to complete the corpus collection and training of the present invention. A 12-core 16G memory plus two NVIDIA 4090 graphics cards with a unit price of several thousand yuan can be used for model training. The above-mentioned large models are all trained using a cluster service architecture, which requires the integration of hundreds, thousands or even tens of thousands of NVIDIA A100 graphics cards. Each A100 graphics card costs tens of thousands of yuan, so the cost is relatively reduced.

[0178] 2. Compared with other large language models, the final model size of the present invention has very low hardware requirements. Compared with ChatGLM-6B with lower configuration requirements, its total model size is 20G, and theoretically requires a graphics card with at least 6G video memory to barely run, which is still too large. The present invention can run smoothly without a PC equipped with a GPU, and the hardware requirements only require a PC with 4 cores and 8G to complete the reasoning task.

[0179] 3. Compared with other large language models, the training of the original corpus in the present invention can be localized anytime and anywhere.

[0180] 4. The unique corpus annotation method and grouping improvement method of the present invention enable accurate reasoning of specialized words even when the corpus is not as large as the above-mentioned large language model, thereby highlighting the accuracy and flexibility of the model.

[0181] 5. The present invention adopts a mixed training mode of Chinese and minority languages ​​and possesses the minority language reasoning capability which the above-mentioned professional direction models do not have.

[0182] 6. The training parameter setting matching of this model can effectively compress the size of the model.

[0183] 7. The text generation output part of this model adopts a unique context detection mechanism that can effectively detect the relevance of the context, effectively improving the output quality of text generation.

[0184] 8. This model adopts a unique fuse mechanism, which can effectively fuse the part of the generated text that does not meet the requirements of the question, effectively shield the sentences that do not meet the requirements of the question, and improve the output quality.

[0185] 9. The present invention further subdivides the professional field into more detailed professional field directions, constructs professional subdivision field models, and adopts multi-model parallel reasoning, which is more professional and more in-depth in this professional field than other models.

[0186] 10. The present invention is a professional-grade lightweight model specially built for the agricultural field. Compared with other models, it has the characteristics of small size, fast inference speed, high quality of inference professional knowledge, more flexible matching, extremely low hardware requirements, and support for Chinese and small languages.

[0187] Device Example 1

[0188] An embodiment of the present invention provides an electronic device, including:

[0189] processor; and,

[0190] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps as described in the above method embodiment.

[0191] Device Example 2

[0192] An embodiment of the present invention provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps described in the above method embodiment.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large language model training method in the agricultural field, characterized in that include: Collecting agricultural data and book information, and then preprocessing to obtain a first Chinese data set and a first minority language data set, and merging the first Chinese data set and the first minority language data set to generate a first mixed language data set; After the first mixed language data set is processed by using a word segmentation library, a preset special character processing method is used for secondary processing, and a word segmentation table is constructed, and a WordPiece word piece model is constructed to process and divide the word segmentation table; Use word embedding algorithm to convert each word in the word segmentation table into a word vector; Construct an encoder component and a decoder component based on the Transformer architecture with a self-attention mechanism to generate a large language model in the agricultural field, and, The agricultural field large language model is trained using word vectors as input to obtain a trained agricultural field large language model.

2. The method according to claim 1, characterized in that The step of collecting agricultural data and book information and preprocessing to obtain the first Chinese data set and the first minority language data set specifically includes: Collecting data and book information in the agricultural field, classifying them according to the fields involved, and then converting the text to obtain a first data set; Cleaning the first data set and removing garbled characters to obtain a second data set; Convert the second data set into simplified Chinese characters to obtain a third data set; Merging the third data set based on the knowledge graph structure to obtain a first Chinese data set; Translating the first Chinese data set into a minority language and performing grammar correction thereon to obtain a first minority language data set; The first Chinese data set and the first minority language data set are merged according to classification to generate a first mixed language data set.

3. The method according to claim 1, characterized in that The preset special character processing method specifically includes: Secondary processing is performed on special tags that cannot be processed by the word segmentation library, including: Customize special character dictionaries based on the characteristics of the Chinese language and minority languages; Get the file path that needs secondary word segmentation and the reading of the custom special character dictionary; The specified file is read according to the file path that needs secondary word segmentation, the word segmentation is performed using the word segmenter, and the words in the custom special character dictionary are considered to build a dictionary that indexes the words to the positions where they appear in the text.

4. The method according to claim 1, characterized in that: The construction of the WordPiece word piece model to process and divide the word segmentation table specifically includes: Loading the vocabulary table specifically includes: extracting all words in the word segmentation table as basic data for training the WordPiece model; Data set division, specifically including: dividing the words in the word segmentation table into a training set and a test set according to a preset ratio; Training a WordPiece model specifically includes: defining a Tokenizer instance and configuring WordPiece model parameters, then training the WordPiece model with a training set through an iterator, and specifying a vocabulary size and a minimum word frequency, and finally, saving the trained WordPiece model to a specified directory, wherein the WordPiece model parameters include special symbols and a maximum input character length.

5. The method according to claim 1, characterized in that The encoding component and the decoder component using the self-attention mechanism based on the Transformer architecture specifically include: Construct an encoder with a preset number of layers, each of which includes: a self-attention layer and a feedforward network; Construct a decoder with a preset number of layers, each of which includes: a self-attention layer, an attention layer, and a feedforward network; The self-attention layer uses a self-attention mechanism to make the model pay attention to the word at the current position and the words at other positions in the sentence; The attention layer causes the decoder to focus on relevant parts of the input sentence; The feedforward network layer is a fully connected feedforward network. The words at each position pass through this identical feedforward neural network separately. The feedforward network layer consists of two linear transformations, i.e., two fully connected layers. The activation function of the first fully connected layer is the ReLU activation function. In each encoder and decoder, the feedforward network layer has the same structure, but does not share parameters. Among them, the number of layers of the encoder is equal to the number of layers of the decoder.

6. The method according to claim 1, characterized in that The word vector is used as input to train the agricultural field large language model to obtain a trained agricultural field large language model, which specifically includes: Data loading and batch processing, specifically including: loading training data and test data separately, and the batch processing function randomly selects data fragments as batches to ensure that each iteration of the model can receive different parts of the data; The training loop includes: using the AdamW optimizer to improve the generalization ability of the model by separating weight decay. In each iteration, the code performs the standard process of zero gradient zeroing, forward propagation, loss calculation, back propagation, and gradient update; Model saving and loading, specifically including: saving model parameters after training.

7. The method according to claim 1, characterized in that The self-attention layer in the encoder further adopts a multi-head attention mechanism, in which each group of attention is used to map the input to a different sub-representation space, so that the model pays attention to different positions in different sub-representation spaces.

8. The method according to claim 1, characterized in that The large language model in the agricultural field further adopts a fuse mechanism to improve the response quality of the large language model in the agricultural field by calculating the cosine similarity between paragraphs output by the model.

9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the large language model training method in the agricultural field as described in any one of claims 1-8.

10. A storage medium for storing computer executable instructions, which, when executed, implement the steps of the large language model training method in the agricultural field as described in any one of claims 1 to 8.

Images