Method for performing time cognition completion on large model by using BERT model

Abstract

A method for performing time cognition completion on a large model by using a BERT model comprises the following steps: performing time extraction on a question of a user through a small model to obtain date and time nouns; and converting the date and the time noun into specific time or date by using post-processing logic, supplementing the specific time or date to a user question, and sending a complemented result into a large model for reasoning. A Bert-base model is selected as a small model. The method comprises the following steps: step 1, defining a model structure; step 2, training data preprocessing; step 3, model training; 4, performing model reasoning and post-processing; step 5, complementing time and date; and step 6, large model reasoning. According to the method for performing time cognition completion on the large model by utilizing the BERT model, the problem of poor time reasoning capability of the large model can be solved, and the accuracy and intelligence of the large model on time reasoning are improved.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, natural language processing, and large model technology, and in particular to a method for performing temporal cognition completion on a large model using a BERT model. Background Art

[0002] Large Language Model (LLM), also known as Large Language Model, is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are characterized by their large size, which helps them learn complex patterns in language data. These models are usually based on deep learning architectures, which helps them achieve impressive performance on various NLP tasks.

[0003] The LLM technology is very mature in terms of understanding and generation, but it also has many shortcomings. For example, even if you tell the LLM what the current time is, it will often give wrong answers when asked about time, such as next Monday, next month, etc.

[0004] Therefore, in view of the shortcomings of the existing technology, it is necessary to provide a method for using the BERT model to perform temporal cognition complement on large models to overcome the shortcomings of the existing technology. Summary of the invention

[0005] The purpose of the present invention is to avoid the shortcomings of the prior art and provide a method for using the BERT model to complete the temporal cognition of a large model, thereby making up for the problem of poor temporal reasoning ability of the large model and improving the accuracy of temporal reasoning of the large model.

[0006] The purpose of the present invention is achieved through the following technical measures.

[0007] Provides a method for using the BERT model to complete the time cognition of the large model. The Bert-base model is used to extract the time of the user's question to obtain the date and time nouns. Then, using post-processing logic, we convert date and time nouns into specific times or dates, add them to the user’s questions, and send the completed results to the big model for reasoning.

[0008] Preferably, the method of using the BERT model to complete the time cognition of the large model is performed by the following steps: Step 1. Define the model structure; Step 2. Preprocessing of training data; Step 3. Model training; Step 4. Model reasoning and post-processing; Step 5. Complete the time and date; Step 6. Large model reasoning.

[0009] Preferably, in the above method of using the BERT model to complete the temporal cognition of the large model, the definition of the model structure in step 1 specifically includes: selecting the Bert-base model as the small model, For the Bert-base model, in order to extract the date and time information in the text, the following definition is made: Define the extraction categories as: date, time; Define input as: text; The output is defined as: the category information corresponding to each word in the text; The categories are defined as follows: irrelevant words are 0, the beginning of a date noun is 1, the middle part of a date noun is 2, the beginning of a time noun is 3, and the middle part of a time noun is 4; Set the meaning of the model output layer to change the category label.

[0010] Preferably, in the above method of using the BERT model to complete the temporal cognition of the large model, step 2. training data preprocessing specifically includes: 2.1. Training data preparation, specifically preparing data in the corpus format supported by the Bert-base model, annotating the corpus text used as training data by category, and storing it in a jsonL file; src represents the original corpus text, and target represents the annotation; 2.2. Data preprocessing, specifically setting the maximum length of text to 256, and processing each piece of data; if the text exceeds 256 characters, it will be truncated directly; if the text is less than 256 characters, it will be padded with 0; Use the BertTokenizer class in the transformers library to load the tokenizer. When tokenizing, each word is segmented separately to obtain the data set required for model training.

[0011] Preferably, in the above method of using the BERT model to complete the temporal cognition of a large model, the model training in step 3 specifically uses the transformers framework to train the model.

[0012] Furthermore, first load the model weights and set the classification labels; then use dataloader to load the dataset in step 2, set the training batch data, learning rate, and maximum number of training; finally, save the training model weights in the get-date-time directory.

[0013] Better yet, load the default model weights, set the number of class labels to 5, the number of training batches to 32, the learning rate to 0.00005, and train for a maximum of 5 epochs.

[0014] Preferably, in the above method of using the BERT model to complete the time cognition of the large model, the post-processing of step 4 specifically includes: intercepting the corresponding time text information from the inference result obtained by the model; and finally converting the corresponding date or time into the accurate time according to the current time.

[0015] Preferably, in the above method of using the BERT model to complete the time cognition of the large model, the time and date completion in step 5 specifically involves adding the specific date or time in brackets after the corresponding time or date in the text.

[0016] Preferably, the above method of using the BERT model to complete the time cognition of the large model sends the result of the completion in step 5 to the large model for reasoning.

[0017] The present invention uses a BERT model to complete the time cognition of a large model. The small model extracts time from the user's question to obtain date and time nouns; the post-processing logic is used to convert the date and time nouns into specific time or date, add them to the user's question, and send the completed results to the large model for reasoning. The BERT (Bidirectional Encoder Representations from Transformers) model, also known as the Bert-base model, is a pre-trained language representation model based on the Transformer architecture, which has made breakthrough progress in the field of natural language processing (NLP). The core advantage of the Bert-base model is that it can learn the bidirectional context information of the language from a large-scale text corpus through pre-training. The present invention uses a BERT model to complete the time cognition of a large model. The BERT model extracts time from the user's question to obtain date and time related nouns. The post-processing logic is used to convert the date and time nouns into specific time or date, add them to the user's question, and make up for the problem of poor time reasoning ability of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0019] Figure 1 It is a flowchart of a method of using the BERT model to perform temporal cognition completion on a large model in the present invention.

[0020] Figure 2It is a schematic diagram of the result of changing the meaning of the output layer of the Bert model to the corresponding category label in the method of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described with reference to the following examples.

[0022] Example 1 A method for using the BERT model to complete the time cognition of a large model. The small model extracts the time from the user's question to obtain the date and time nouns. Then, the post-processing logic is used to convert the date and time nouns into specific time or date, which is then added to the user's question. The completed result is sent to the large model for reasoning.

[0023] The method of using the BERT model to complete the time cognition of the large model is carried out through the following steps: Step 1. Define the model structure; Step 2. Preprocessing of training data; Step 3. Model training; Step 4. Model reasoning and post-processing; Step 5. Complete the time and date; Step 6. Large model reasoning.

[0024] In order to improve the accuracy of time cognition of the large model LLM, the model structure defined in step 1 specifically includes: selecting the Bert-base model as the small model. The BERT (Bidirectional Encoder Representations from Transformers) model, also known as the Bert-base model, is a pre-trained language representation model based on the Transformer architecture, which has made breakthrough progress in the field of natural language processing (NLP). The core advantage of the Bert-base model is that it can learn the bidirectional context information of the language from a large text corpus through pre-training. The present invention uses the BERT model to complete the time cognition of the large model, and extracts the time of the user's question through the BERT model to obtain the nouns related to date and time. Then, the post-processing logic is used to convert the date and time nouns into specific time or date, and supplement them into the user's question to make up for the problem of poor time reasoning ability of the large model. The structure of the bert-base model is common knowledge in the field and will not be repeated here. The bert-base model is pre-trained on tens of millions of supervised data. Therefore, it has better support for Chinese semantic understanding. Since the training tasks of the pre-trained model itself have nothing to do with information extraction, downstream training tasks need to be formulated based on the model.

[0025] Specific task: To complete the date and time of a text, it is necessary to accurately extract the date and time information in the text. For the Bert-base model, in order to extract the date and time information in the text, the following definition is made: Define the extraction categories as: date, time; Define input as: text; The output is defined as: the category information corresponding to each word in the text; The categories are defined as follows: irrelevant words are 0, the beginning of a date noun is 1, the middle part of a date noun is 2, the beginning of a time noun is 3, and the middle part of a time noun is 4; Set the meaning of the model output layer to change the category label.

[0026] For example, for my task, I need to change the meaning of the model output layer to category labeling, and the corresponding results are as follows Figure 2 shown.

[0027] My task is: "Have a meeting at 8 o'clock tonight", and the output category is labeled: "12344400".

[0028] Step 2. Training data preprocessing specifically includes: 2.1. Training data preparation, specifically preparing data in the corpus format supported by the Bert-base model, annotating the corpus text used as training data, and storing it in a jsonL file. src represents the original corpus text, and target represents the annotation.

[0029] For example: {"src":"Meeting at 8 o'clock tonight","target":"12344400"}.

[0030] 2.2. Data preprocessing, specifically setting the maximum length of text to 256, and processing each piece of data; if the text exceeds 256 characters, truncate it directly; for text less than 256 characters, fill it with 0.

[0031] Use the BertTokenizer class in the transformers library to load the tokenizer: from transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained('bert-base-chinese') Since each word needs to be marked, do not use the default word segmentation method when segmenting words, but segment each word separately.

[0032] Then get the data set required for model training, such as input_ids, attention_mask, labels, etc.

[0033] The model training in step 3 uses the transformers framework to train the model. First, load the model weights and set the classification labels; then use the dataloader to load the dataset in step 2, set the training batch data, learning rate, and maximum number of training; finally, save the training model weights in the get-date-time directory.

[0034] Then proceed to step 4 and use the new model for inference.

[0035] Input text: I will still be here waiting for you this time next year; Output: 120120000000.

[0036] The post-processing of step 4 specifically includes: extracting the corresponding time text information from the inference results obtained by the model; and finally converting the corresponding date or time into the accurate time according to the current time.

[0037] For example, if we extract the corresponding text from the inference results obtained by the model, we can get ["next year", "today"].

[0038] The last step of post-processing is to convert the corresponding date or time into the accurate time according to the current time. This step can be implemented directly by writing a program.

[0039] For example, if the current time is 2024-10-31 18:38:00, then we get that this year is 2024, and we encode it in the program logic to get the dates of next year, the year after, the year after, etc. Similarly, we can get the dates of today, tomorrow, and the day after tomorrow.

[0040] Step 5 of completing the time and date is to add the specific date or time in brackets after the corresponding time or date in the text.

[0041] Finally, proceed to step 6, large model reasoning. The completed result of step 5 is sent to the large model for reasoning.

[0042] The present invention uses the BERT model to complete the time cognition of the large model. The BERT model extracts time from the user's question and obtains nouns related to date and time. Then, the post-processing logic is used to convert the date and time nouns into specific time or date and add them to the user's question, which makes up for the poor time reasoning ability of the large model and can improve the time reasoning ability of the large model.

[0043] Example 2 A method for using a BERT model to complete temporal cognition of a large model, the other features of which are the same as those of Example 1, except that: it also has the following specific features.

[0044] Step 1 defines the model structure, specifically including: selecting the Bert-base model as the small model and the large model as the LLM model.

[0045] Step 2. Training data preprocessing 1) Training data preparation The data is annotated in the following format and stored in a jsonL file, where src represents the original text and target represents the annotation.

[0046] {"src":"Meeting at 8pm tonight","target":"12344400"} {"src":"Are you free tomorrow?","target":"012000"} {"src":"Canton Tower will have fireworks show next Wednesday","target":"00012200000"} {"src":"This month's performance is good","target":"12200000"} {"src":"I will try to get into university next year","target":"12000000"} ... You can prepare 10,000 pieces of relevant data. If it is troublesome to do it manually, you can also generate data through some small programs or with the assistance of large models.

[0047] 2) Data preprocessing Set the maximum length of text to 256, and process each piece of data. If the text exceeds 256 characters, it will be truncated directly; if the text is less than 256 characters, it will be padded with 0.

[0048] Use the BertTokenizer class in the transformers library to load the tokenizer: from transformers import BertTokenizer; tokenizer = BertTokenizer.from_pretrained('bert-base-chinese').

[0049] Define special word id: pad id=tokenizer.convert_tokens_to_ids('[PAD]'); cls_id =tokenizer.convert_tokens_to_ids('[CLS]'); sep id=tokenizer.convert_tokens_to_ids('[SEP]'); unk id=tokenizer.convert_tokens_to_ids('[UNK]'); Since each word needs to be marked, do not use the default word segmentation method when segmenting words, but segment each word separately.

[0050] The processing method of this step can be implemented through a program, for example: datasets = [] for text,label in data: input_ids=[cls_id] + [tokenizer.convert_tokens_to_ids(i) for i intext] + [sep_id] # Calculate how many tokens are needed pad_len = 256 - len(input_ids) input_ids =input_ids + [pad_id]*pad_len labels = [unk_id] + [i for i in label] + [unk_id] + [unk_id]*pad_len attention_mask = [1]*len(input_ids) + [0]*pad_len # Add to datasets datasets.append( text, np.asarray(input_ids,dtype='int64'), np.asarray(labels,dtype='int64'), np.asarray(attention_mask,dtype='int64') ) .

[0051] The data is divided into training set and test set in an 8:2 ratio.

[0052] 3) Encapsulate the training set and test set into torch.utils.data.Dataset, as shown in the following example: class MyDataset(Dataset): def __init__(self, datasets): self.datasets = datasets def __len__(self): return len(self.datasets) def __getitem__(self, item): data = self.datasets[item] return { 'text': xxx, 'input_ids': xxx, 'attention_mask': xxx, 'labels': xxx }.

[0053] You can get train_dataset and text_dataset respectively.

[0054] Step 3, model training. The model training specifically uses the transformers framework to train the model. First, load the default model weights and set the classification label to 5. Then use dataloader to load the dataset in step 2, set the training batch size to 32, set the learning rate to 0.00005, and train for a maximum of 5 epochs. The final training model weights are saved in the get-date-time directory.

[0055] Model training is carried out through the following procedures: # 2. Load the pre-trained BERT model model = BertForTokenClassification.from_pretrained( 'bert-base-chinese', num_labels=5, # We have 5 tags output_attentions=False, output_hidden_states=False,) # 3. Load data using DataLoader dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True) # 4. Define optimizer and loss function optimizer = AdamW(model.parameters(), lr=5e-5, eps=1e-8) total_steps = len(dataloader) * epochs scheduler = get_linear_schedule_with_warmup( optimizer, num_warmup_steps=0, num_training_steps=total_steps) # 5. Training the model device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model.to(device) model.train()for epoch in range(epochs): for batch in dataloader: batch = {k: v.to(device) for k, v in batch.items()} optimizer.zero_grad() outputs = model( input_ids=batch['input_ids'], attention_mask=batch['attention_mask'], labels=batch['labels'] ) loss = outputs.loss loss.backward() optimizer.step() scheduler.step() After training for 5 rounds, save the model and get a new model.

[0056] Then proceed to step 4, model reasoning and post-processing.

[0057] 1) Model Reasoning Use the new model for inference. For example: Input text: I will still be here waiting for you this day next year, output result: 120120000000.

[0058] 2) Post-processing: Call the post-processing program logic to get next year = 2025; today = 2024-10-31.

[0059] Step 5: Completing the time and date: Specifically, add the specific date or time in brackets after the corresponding time or date in the text.

[0060] For example: Q (text): I will still be here waiting for you this time next year; After completion: I will still be waiting for you here today (2024-10-31) next year (2025).

[0061] Finally, proceed to step 6, large model reasoning. The completed result of step 5 is sent to the large model for reasoning.

[0062] The completed text "I will still be here waiting for you today (2024-10-31) next year (2025)" is input into the big model, and the output of the big model is "Today in 2025, that is, October 31, 2025, will be Friday. If you mean the day of the week of the same day next year, it is now certain that it is this day. If you have other specific questions or need more details, please let me know! ".

[0063] The method of the present invention has two highlights. Highlight 1: The date and time extraction model is trained based on the Bert series model. Highlight 2: Based on the combination of the small model and the large model, the shortcomings of the large model in terms of insufficient time cognition are made up. The small model has a fast reasoning speed and has little impact on the large model question and answer.

[0064] The present invention uses the BERT model to complete the time cognition of the large model. The BERT model extracts time from the user's question and obtains nouns related to date and time. Then, the post-processing logic is used to convert the date and time nouns into specific time or date and add them to the user's question, which makes up for the poor time reasoning ability of the large model and can improve the time reasoning ability of the large model.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for using the BERT model to complete the temporal cognition of a large model, characterized in that: Use the Bert-base model to extract time from user questions and obtain date and time nouns; Then, using post-processing logic, we convert date and time nouns into specific times or dates, add them to the user’s questions, and send the completed results to the big model for reasoning.

2. The method for performing temporal cognition completion on a large model using a BERT model according to claim 1, characterized in that: Proceed as follows: Step 1. Define the model structure; Step 2. Preprocessing of training data; Step 3. Model training; Step 4. Model reasoning and post-processing; Step 5. Complete the time and date; Step 6. Large model reasoning.

3. The method for performing temporal cognition completion on a large model using a BERT model according to claim 1, characterized in that: Step 1 defines the model structure, including: For the Bert-base model, in order to extract the date and time information in the text, the following definition is made: Define the extraction categories as: date, time; Define input as: text; The output is defined as: the category information corresponding to each word in the text; The categories are defined as follows: irrelevant words are 0, the beginning of a date noun is 1, the middle part of a date noun is 2, the beginning of a time noun is 3, and the middle part of a time noun is 4; Set the meaning of the model output layer to change the category label.

4. The method for performing temporal cognition completion on a large model using a BERT model according to claim 3, characterized in that: Step 2. Training data preprocessing specifically includes: 2.

1. Training data preparation, specifically preparing data in the corpus format supported by the Bert-base model, annotating the corpus text used as training data by category, and storing it in a jsonL file; src represents the original corpus text, and target represents the annotation; 2.

2. Data preprocessing, specifically setting the maximum length of text to 256, and processing each piece of data; if the text exceeds 256 characters, it will be truncated directly; if the text is less than 256 characters, it will be padded with 0; Use the BertTokenizer class in the transformers library to load the tokenizer. When tokenizing, each word is segmented separately to obtain the data set required for model training.

5. The method for performing temporal cognition completion on a large model using a BERT model according to claim 4, characterized in that: The model training in step 3 specifically uses the transformers framework to train the model.

6. The method for performing temporal cognition completion on a large model using a BERT model according to claim 5, characterized in that: First, load the model weights and set the classification labels. Then use dataloader to load the dataset in step 2, set the training batch data, learning rate, and maximum number of training samples. Finally, save the training model weights in the get-date-time directory.

7. The method for performing temporal cognition completion on a large model using a BERT model according to claim 6, characterized in that: Load the default model weights, set the classification label to 5, set the training batch size to 32, set the learning rate to 0.00005, and train for a maximum of 5 epochs.

8. The method for performing temporal cognition completion on a large model using a BERT model according to any one of claims 5 to 7, characterized in that: The post-processing of step 4 specifically includes: extracting the corresponding time text information from the inference results obtained by the model; and finally converting the corresponding date or time into the accurate time according to the current time.

9. The method for performing temporal cognition completion on a large model using a BERT model according to claim 8, characterized in that: Step 5 of completing the time and date is to add the specific date or time in brackets after the corresponding time or date in the text.

10. The method for performing temporal cognition completion on a large model using a BERT model according to claim 9, characterized in that: The completed result of step 5 is sent to the large model for reasoning.

Images