Document grading title generation method and device based on large language model

Abstract

The embodiment of the invention provides a document hierarchical title generation method and device based on a large language model. The method comprises the steps that a long text needing to generate a hierarchical title and a preset prompt statement are received; splicing the long text and the prompt statement to obtain a complete input text; converting the input text into an input feature; processing the input features by using a large model for training fine tuning to obtain a target text with graded title information; and matching the target text into the long text and outputting the target text. According to the method, the technical problem that in the prior art, troubles of different degrees exist when an author reviews and arranges text content or other readers retrieve effective information is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating document hierarchical titles based on a large language model, a device for generating document hierarchical titles based on a large language model, an electronic device, and a computer-readable medium. Background Art

[0002] Large language models are usually based on deep learning architectures, such as the Transformer structure that is leading the era in the field of natural language processing. They have a huge number of parameters, most of which have billions, tens of billions, or even hundreds of billions. By training on a huge amount of text corpus, they have good generalization and universal capabilities and perform well in downstream tasks (such as text summarization, text translation, sentiment analysis, etc.). They can support long texts in terms of input and output, which also enables large language models to handle various tasks, including complex learning and reasoning.

[0003] When authors write long articles such as essays and blogs, they often do not organize the perfect logic in advance, and important information points are distributed in long chapters. For ordinary readers, it is difficult to quickly grasp the key points of each module when reading a long document without hierarchical titles, and reading the whole article directly is undoubtedly time-consuming. A long document without hierarchical titles will cause varying degrees of trouble for the author to review and organize the text content, or for other readers to retrieve effective information.

[0004] Therefore, it is necessary to refine and grade the titles of such long chapter texts. Summary of the invention

[0005] The embodiment of the present invention provides a method and device for generating document hierarchical titles based on a large language model to solve the technical problem in the prior art that there are varying degrees of difficulties for the author to review and organize the text content, or for other readers to retrieve valid information.

[0006] The embodiment of the present invention discloses a method for generating document hierarchical titles based on a large language model, comprising:

[0007] Receive the long text for which a hierarchical title needs to be generated and a pre-set prompt sentence;

[0008] The long text and the prompt sentence are concatenated to obtain a complete input text;

[0009] Converting the input text into input features;

[0010] Process the input features using the trained and fine-tuned large model to obtain target text with graded title information;

[0011] The target text is matched to the long text and outputted.

[0012] Optionally, the long text is a document with more than 500 words and more than one paragraph;

[0013] The target text includes at least two levels of titles: each level of title format is: line number, title level, title content;

[0014] The input features include: input_ids array and attention_mask array.

[0015] Optionally, the long text and the prompt statement are concatenated to obtain a complete input text, including:

[0016] According to the information of the splicing in the training phase, the prompt sentence is inserted into the designated position of the long text to obtain the complete input text;

[0017] The prompt statement includes: generating a hierarchical title, outputting an insertion position, a title level, and a title content.

[0018] Optionally, the input features are processed using a trained and fine-tuned large model to obtain a target text with hierarchical title information, including:

[0019] Input the input features into the trained and fine-tuned large model, which predicts the token of the next position. Each time, the token with the highest probability at that position is taken and concatenated to the input features, and the attention_mask is modified accordingly. Repeat the above process until the predicted token is the termination symbol. <eos>, get the predicted sequence;

[0020] The prompt sentence and the original text are removed from the prediction sequence to obtain the target text.

[0021] Optionally, the method further comprises:

[0022] According to a pre-given training data set, a large language model is trained to obtain a large model for training and fine-tuning;

[0023] The training and fine-tuning large model includes: at least two categories of sub-models;

[0024] During the training process, the sub-model of each category is trained using the long document training data of that category;

[0025] Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including:

[0026] According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

[0027] Optionally, after receiving the long text for which a hierarchical title needs to be generated and a preset prompt statement, the method further includes:

[0028] Using a classification model to classify the long text to obtain category information;

[0029] Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including:

[0030] According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

[0031] This embodiment also discloses a document hierarchical title generation device based on a large language model, comprising:

[0032] A receiving module, used for receiving a long text for which a hierarchical title needs to be generated and a preset prompt sentence;

[0033] A splicing module, used for splicing the long text and the prompt sentence to obtain a complete input text;

[0034] A conversion module, used for converting the input text into input features;

[0035] A target text generation module, used to process the input features using the trained and fine-tuned large model to obtain a target text with hierarchical title information;

[0036] An output module is used to match the target text into the long text and output it.

[0037] Optionally, the device for generating document hierarchical titles based on a large language model further includes:

[0038] A large model training module is used to train a large language model based on a pre-given training data set to obtain a large model for training and fine-tuning;

[0039] The training and fine-tuning large model includes: at least two categories of sub-models;

[0040] During the training process, the sub-model of each category is trained using the long document training data of that category;

[0041] The target text generation module is specifically used to process the input features according to the category information of the input features using the sub-model corresponding to the category information to obtain the target text with hierarchical title information.

[0042] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0043] The memory is used to store computer programs;

[0044] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0045] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0046] The embodiments of the present invention include the following advantages:

[0047] In the embodiment of the present invention, a powerful large language model technology is used to generate hierarchical titles based on the creative content, which is more conducive to obtaining hierarchical titles with clear content division and clear structure.

[0048] Specifically, the embodiment of the present invention integrates multiple generation targets such as title insertion position, title level, and title content information into a single target text segment through a specific format, and uses a large generative model to perform overall prediction at one time. There is no need to disassemble the tasks, and serial output is achieved one by one, thus achieving end-to-end result generation and effectively improving performance.

[0049] At the same time, the embodiment of the present invention uses appropriate prompt statements to activate the memory of the large model pre-training stage, and only a small amount of fine-tuning data is needed to fine-tune the model, thereby improving the performance of the model.

[0050] Furthermore, in the model fine-tuning evaluation stage, the embodiment of the present invention uses the accuracy rate as the evaluation index for the title insertion position and title level output by the model, and uses BLUE as the evaluation index for the title content output by the model. By using the evaluation index to evaluate the three different tasks respectively, the performance of the fine-tuned model can be evaluated more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of the steps of a method for generating document hierarchical titles based on a large language model provided in an embodiment of the present invention;

[0052] Figure 2 It is a structural block diagram of a document hierarchical title generation device based on a large language model in an embodiment of the present invention;

[0053] Figure 3 is an overall flow chart provided in an embodiment of the present invention;

[0054] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Large Language Model (LLM) is an artificial intelligence model that can give the required results in text form based on the input natural language text and prompt sentences.

[0058] The embodiment of the present invention provides a method for generating hierarchical titles of documents based on a large language model, which can use the text generation technology based on the large language model to generate titles at all levels, and simultaneously realize the positioning of the title position and the determination of the title level. In the embodiment of the present invention, the powerful large language model technology is used to generate hierarchical titles based on the creative content, which is more conducive to obtaining hierarchical titles with clear content division and clear structure.

[0059] The hierarchical title generation for long documents aims to generate first-level titles, second-level titles, etc. for a long document without subdivided subheadings. Generating hierarchical titles for a long document is helpful to clarify the structure and hierarchy of the document, sort out the logic of the article, clarify the core content of each section, and assist in completing knowledge induction, so as to further build a knowledge graph.

[0060] Reference Figure 1 and Figure 3 , shows a document hierarchical title generation method based on a large language model provided in an embodiment of the present invention, including:

[0061] 101. receiving a long text for which a hierarchical title needs to be generated and a preset prompt statement;

[0062] 102. Concatenate the long text and the prompt sentence to obtain a complete input text;

[0063] 103. Convert the input text into input features;

[0064] 104. Process the input features using the trained and fine-tuned large model to obtain target text with graded title information;

[0065] 105. Match the target text to the long text and output it.

[0066] The long text is a document with more than 500 words and more than one paragraph;

[0067] The target text includes at least two levels of titles: each level of title format is: line number, title level, title content;

[0068] The input features include: input_ids array and attention_mask array.

[0069] The input_ids array is an array of integers representing the tokenized version of the input text. Each integer is an index in the vocabulary corresponding to a specific word or subword (depending on the word segmentation strategy used). For example, if a byte pair encoding (BPE)-based tokenizer is used, a word may be split into multiple subwords. Each word or subword is mapped to a unique integer identifier, which constitutes the input_ids array.

[0070] Similarly, the attention_mask array is also an integer array, usually with the same length as input_ids. Its role is to tell the model which positions of the tokens are valid and which are filled. When processing text sequences of different lengths, the shorter sequences are padded to a fixed length. In order to prevent the model from wasting attention resources on these padded positions, the attention_mask array is used to shield these positions. Usually, the positions of valid tokens are set to 1, and the padded positions are set to 0.

[0071] In this embodiment, converting the input text into input features specifically includes:

[0072] In the BERT model and its derivatives, the input text first goes through a tokenization process, where the text is broken down into words or subwords, and each token is then mapped to a unique integer identifier, which makes up the input_ids array, which represents the digitized form of the text. To accommodate the needs of model processing, the length of the input_ids array is normalized to a fixed value. During this normalization process, inputs that exceed the predetermined value are truncated, and inputs that are shorter than this length are padded by adding specific padding markers. This processing mechanism ensures the consistency of the model input and allows the model to process text data of different lengths in batches.

[0073] For example, assuming the input text is "Hello, world", this sentence needs to be converted into a digital representation that the model can understand. Assume that the word segmenter divides "Hello," into

[7592] and "world" into

[2088] , and the special marker [CLS] (indicating the start of input) has an ID of

[101] and an ID of [SEP] (indicating the end of input) of

[102] . It can be seen that the corresponding input_ids array is: [101, 7592, 2088, 102]. Fill it to a fixed length. Assuming the fixed length is 10, the corresponding input_ids array needs to be extended to a length of 10. This is achieved by adding the [PAD] marker. The extended input_ids array is: [101, 7592, 2088, 102, 0, 0, 0, 0, 0].

[0074] For the expanded input_ids array, the corresponding attention_mask array is: [1,

[0075] It can be seen that the input_ids array provides an efficient digital representation of the input text, while the attention_mask array ensures that the model can effectively identify and focus on the substantive content when processing this digital information, excluding irrelevant padding effects.

[0076] The input_ids array and the attention_mask array together form the basis for the model to process text information, which is crucial to ensuring the performance and analysis accuracy of the model.

[0077] In this embodiment, the long text and the prompt sentence are concatenated to obtain a complete input text, including:

[0078] According to the information of the splicing in the training phase, the prompt sentence is inserted into the designated position of the long text to obtain the complete input text;

[0079] Insert the prompt statement into the specified position of the long text to obtain the complete input text, which specifically includes:

[0080] First, check the format of the long text. If the long text has format problems, such as extra blank characters, special symbols, etc., preprocessing is required. For example, you can use text processing functions to remove extra spaces, line breaks, etc., and organize the long text into a clean and regular form.

[0081] Also check the format of the prompt statement to make sure there are no grammatical errors or inappropriate punctuation. If the prompt statement is user-entered, it may need some simple cleaning, such as removing extra spaces at the beginning and end.

[0082] If the long text is an article and you want to insert a prompt statement after the first paragraph of the article, you can first determine the end position of the first paragraph by finding a line break or a specific paragraph mark, and then use that position as the insertion point for the prompt statement.

[0083] If the long text is stored in a file, you need to read the long text into memory for processing. You can use the file read operation to read the long text line by line into a string or string array.

[0084] After determining the insertion point, modify the long text in memory and insert the prompt statement. Then write the modified content back to the file. It should be noted that when writing the file back, it should be processed according to the original format of the file, such as retaining the original line break and other format information.

[0085] The prompt statement includes: generating a hierarchical title, outputting an insertion position, a title level, and title content. The prompt statement in this embodiment can be a pre-designed statement, and does not require any processing such as user input and editing.

[0086] In a specific implementation process, long text refers to the main text content that needs to be processed or generated, for example, it can be an article, a conversation, a story, etc.

[0087] Hint statements are additional instructions or information provided to the model to guide how the model processes long texts.

[0088] In this embodiment, the input features are processed using a trained and fine-tuned large model to obtain target text with hierarchical title information, including:

[0089] Input the input features into the trained and fine-tuned large model, which predicts the token of the next position. Each time, the token with the highest probability at that position is taken and concatenated to the input features, and the attention_mask is modified accordingly. Repeat the above process until the predicted token is the termination symbol. <eos>, get the predicted sequence;

[0090] The prompt sentence and the original text are removed from the prediction sequence to obtain the target text.

[0091] Specifically, the large model makes predictions based on its internal neural network architecture. Taking the Transformer architecture as an example, the multi-head attention mechanism in the model pays attention to each token in the input feature from different angles to capture the semantic relationship between them.

[0092] Then, the attention-processed information is further transformed and processed through a feedforward neural network.

[0093] When predicting the token at the next position, the large model will calculate the probability distribution of each possible token appearing at the current position based on the learned language patterns, semantic information, etc. This probability distribution is based on the model's learning of a large amount of text data (the data used in the training and fine-tuning stages). For example, for a natural language generation task, if the input feature is "today's weather", the model may calculate the probability of different tokens such as "sunny", "cloudy", and "raining" appearing at this position based on existing knowledge.

[0094] The large model learns a wide range of language knowledge during the pre-training phase, including the co-occurrence relationship of vocabulary, grammatical structure, etc. During the fine-tuning phase, parameters are adjusted for specific tasks or fields so that the model can better adapt to task requirements when predicting the next token. For example, in a large model that has been fine-tuned for news generation, when features related to financial news are input, the model will use the financial vocabulary and expression patterns learned during the fine-tuning phase to predict the next token.

[0095] After the big model predicts the probability distribution of the token at the next position, the token with the highest probability is selected. This selection process is based on the model's understanding of the current input features and existing knowledge. For example, if at a certain position, the probability of the token "beautiful" is 0.3, the probability of "pretty" is 0.25, and the probabilities of other tokens are lower, then "beautiful" will be selected as the token for this position.

[0096] The selected token is concatenated to the end of the input feature. This allows the input feature to grow continuously and gradually build a complete prediction sequence. For example, if the initial input feature is "spring" and the next token selected is "of", then the new input feature becomes "spring of".

[0097] After splicing a new token to the input feature, the attention_mask needs to be modified accordingly. For example, if the original attention_mask indicates attention to the first three tokens of the input feature, after splicing a new token, the attention_mask needs to be adjusted so that it can correctly represent the attention to the new input feature (including the newly added token). This may involve adding an element to the mask, or adjusting the value of an existing element to reflect the change in the input feature.

[0098] Repeat the previous steps, that is, the large model predicts the token at the next position, selects the token with the highest probability, concatenates the token to the input feature, and then modifies the attention_mask. Each iteration will increase the input feature by one token, gradually building a complete prediction sequence.

[0099] In each iteration, the model makes predictions based on the currently updated input features. For example, as the iterations progress, the input features may change from "spring scenery" to "spring scenery is very", then to "spring scenery is beautiful", etc. The model will predict the next token based on each updated input feature.

[0100] Continue iterating until the predicted token is the termination symbol. It indicates the end of a sequence. When the model predicts this symbol, it means that the predicted sequence is complete. For example, when generating an article, when it is predicted, it means that the article has been generated. At this time, the entire sequence obtained (starting from the initial input features and obtained by multiple concatenation of tokens) is the predicted sequence.

[0101] Furthermore, in the prediction sequence, it is necessary to accurately identify which parts are the prompt sentences and the original text. The prompt sentence includes: generating a hierarchical title, outputting the insertion position, the title level, and the title content. The original text is the long text of the original input. Once the prompt sentence and the original text are identified, they are deleted from the prediction sequence. The remaining part is the target text.

[0102] In a specific implementation process, the method for generating document hierarchical titles based on a large language model further includes:

[0103] According to a pre-given training data set, a large language model is trained to obtain a large model for training and fine-tuning;

[0104] The training and fine-tuning large model includes: at least two categories of sub-models;

[0105] During the training process, the sub-model of each category is trained using the long document training data of that category.

[0106] The large model is composed of at least two categories of sub-models, and the requirements of different categories of input features need to be considered in actual use. For example, for different fields (such as news, scientific articles, novels, etc.), targeted sub-models are required.

[0107] In this embodiment, the pre-given training data set includes long documents of various categories and their corresponding hierarchical titles. The training data set can be divided into the following parts:

[0108] Long documents: the content of each document;

[0109] Prompt statement: Please generate a hierarchical title for the article at the appropriate position, and output the insertion position, title level, title content, etc.;

[0110] Category information: the category to which the document belongs, such as news, scientific articles, novels, etc.;

[0111] Note: Correct classification title and where to insert it.

[0112] Divide the training data set into different categories, each of which corresponds to a sub-model. For example, the data set can be divided into types such as "news", "scientific articles", and "novels".

[0113] The specific training steps are:

[0114] Concatenate long documents and prompt statements into the completed input text, separated by special tags;

[0115] Choose a large language model suitable for the task, such as BERT, GPT, T5, etc.

[0116] Set training parameters, such as learning rate, batch size, number of training rounds, etc.;

[0117] Use the training dataset to train the model to ensure that the model can learn the basic ability to generate rated titles.

[0118] Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including:

[0119] According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

[0120] In this embodiment, after receiving the long text and prompt statement for generating a hierarchical title, the following steps are included:

[0121] Using a classification model to classify the long text to obtain category information;

[0122] The classification model in this embodiment is a naive Bayes classification model or a convolutional neural network classification model in deep learning;

[0123] Among them, the Naive Bayesian classification model is a probabilistic classification method based on the Bayesian theorem, which assumes that the features are independent of each other. This model has a wide range of applications in fields such as text classification. It can process large amounts of text data and has fast calculation speed. The core of the Naive Bayesian model is to calculate the posterior probability of a given data feature and select the category with the largest posterior probability as the prediction result.

[0124] In this embodiment, the long text is classified using a naive Bayes classification model, and obtaining category information specifically includes:

[0125] Calculate the prior probability of each category, that is, the frequency of each category appearing in the training set;

[0126] Calculate the probability of each feature under each class condition, which usually involves calculating the number of times the feature value appears in samples of a specific class divided by the total number of samples of that class;

[0127] For a new sample, calculate the posterior probability of it belonging to each category, that is, the probability of each category under the given sample characteristics, and select the category with the largest posterior probability as the prediction result.

[0128] For example, for a long text, the probability of each word in the text appearing in different categories is counted to calculate the probability that the long text belongs to each category. For example, in news classification, if words such as "stock" and "exchange rate" often appear in news in the "economy" category, when these words appear more frequently in the long text, the probability that the long text belongs to the "economy" category will increase.

[0129] The convolutional neural network classification model in deep learning regards long text as a one-dimensional sequence. The convolution kernel in the convolution layer slides on the text sequence to perform convolution operations, automatically extract local features in the text, and then compress the features through the pooling layer, and finally classify through the fully connected layer.

[0130] Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including:

[0131] According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

[0132] In this embodiment, the sub-model corresponding to the category information is used to process the input feature to obtain the target text with the hierarchical title information, which specifically includes:

[0133] Input the input features into the sub-model under the trained and fine-tuned large model, predict the token of the next position, take the token with the highest probability at that position and append it to the input features each time, and modify the attention_mask accordingly; repeat the above process until the predicted token is the termination symbol <eos>, get the predicted sequence;

[0134] The prompt sentence and the original text are removed from the prediction sequence to obtain the target text.

[0135] In the specific implementation process, after obtaining the target text with hierarchical title information, it is also necessary to post-process the target text to check the accuracy, logic and coherence of the target text with hierarchical title information, to ensure that the hierarchical relationship between the hierarchical titles is correct and can accurately reflect the main content of the input features. If problems are found in the target text, such as unclear semantics or confusing hierarchies, adjustments and optimizations are required.

[0136] Post-processing the target text specifically includes:

[0137] Syntax checking, semantic accuracy checking, logical relationship checking, and redundant information removal.

[0138] Among them, grammar checking is to check whether the target text structure is complete, for example, whether there is a missing subject, predicate or object, to ensure that the sentence structure is complete in the formal target text with hierarchical title information;

[0139] The semantic accuracy check needs to determine whether the concepts in the target text are consistent with the input feature content. For example, if the input feature mainly discusses the application of artificial intelligence in the medical field, and the title contains a statement about artificial intelligence in the financial field, there is a problem of inconsistent concepts, and the hierarchical title information needs to be modified to ensure semantic accuracy;

[0140] Furthermore, it is necessary to check whether the target text fully conveys the key information in the input features. For example, if the input features are about the development process, efficacy and side effects of a new drug, and the title only mentions the development process, then the title lacks semantic completeness and needs to be supplemented with relevant information, such as "New drugs: development process, significant efficacy and potential side effects".

[0141] Logical relationship check is to check the hierarchical relationship and order of the hierarchical title information. For example, if the first-level title is "Plant Growth Process", the second-level titles "Seed Germination", "Seedling Growth", "Plant Maturity", etc. should be reasonable subdivisions of the first-level title. The second-level titles should not be logically disconnected from the first-level titles or have a chaotic hierarchy.

[0142] In addition, if the input features describe the development of events in chronological order, then the titles should also be arranged in chronological order to reflect logical coherence.

[0143] Redundant information removal is to check whether there are repeated expressions in the target text and delete them. For example, "New materials: characteristics and applications of new materials", where "new materials" is repeated, can be modified to "new materials: characteristics and applications";

[0144] At the same time, remove the information in the target text that is irrelevant to the input feature content. For example, if the input feature is about historical events, but the elements of modern pop culture appear in the rating title, this irrelevant content needs to be removed.

[0145] The method of this embodiment provides document creators with a method for generating hierarchical titles, refining and sorting out the content of long documents such as essays and blogs; at the same time, it generates hierarchical titles for a long document without a hierarchical title for readers, so that readers can quickly grasp the key points of each module and save reading time; further, clear and concise hierarchical titles are divided for long documents to assist in knowledge summarization and facilitate further knowledge graph construction.

[0146] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0147] Reference Figure 2 , shows a document hierarchical title generation device based on a large language model provided in an embodiment of the present invention, comprising:

[0148] A receiving module, used for receiving a long text for which a hierarchical title needs to be generated and a preset prompt sentence;

[0149] A splicing module, used for splicing the long text and the prompt sentence to obtain a complete input text;

[0150] A conversion module, used for converting the input text into input features;

[0151] A target text generation module, used to process the input features using the trained and fine-tuned large model to obtain a target text with hierarchical title information;

[0152] An output module is used to match the target text into the long text and output it.

[0153] In this embodiment, the input features are input_ids array and attention_mask array.

[0154] The input_ids array is an array of integers representing the tokenized version of the input text. Each integer is an index in the vocabulary corresponding to a specific word or subword (depending on the word segmentation strategy used). For example, if a byte pair encoding (BPE)-based tokenizer is used, a word may be split into multiple subwords. Each word or subword is mapped to a unique integer identifier, which constitutes the input_ids array.

[0155] Similarly, the attention_mask array is also an integer array, usually with the same length as input_ids. Its role is to tell the model which positions of the tokens are valid and which are filled. When processing text sequences of different lengths, the shorter sequences are padded to a fixed length. In order to prevent the model from wasting attention resources on these padded positions, the attention_mask array is used to shield these positions. Usually, the positions of valid tokens are set to 1, and the padded positions are set to 0.

[0156] A large model training module is used to train a large language model based on a pre-given training data set to obtain a large model for training and fine-tuning;

[0157] The training and fine-tuning large model includes: at least two categories of sub-models;

[0158] During the training process, the sub-model of each category is trained using the long document training data of that category;

[0159] The target text generation module is specifically used to process the input features according to the category information of the input features using the sub-model corresponding to the category information to obtain the target text with hierarchical title information.

[0160] In this embodiment, a classification model is used to classify the long text to obtain category information;

[0161] The classification model is a naive Bayes classification model or a convolutional neural network classification model in deep learning;

[0162] Among them, the Naive Bayesian classification model is a probabilistic classification method based on the Bayesian theorem, which assumes that the features are independent of each other. This model has a wide range of applications in fields such as text classification. It can process large amounts of text data and has fast calculation speed. The core of the Naive Bayesian model is to calculate the posterior probability of a given data feature and select the category with the largest posterior probability as the prediction result.

[0163] The convolutional neural network classification model in deep learning regards long text as a one-dimensional sequence. The convolution kernel in the convolution layer slides on the text sequence to perform convolution operations, automatically extract local features in the text, and then compress the features through the pooling layer, and finally classify through the fully connected layer.

[0164] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0166] Memory 1303, used for storing computer programs;

[0167] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0168] Receive the long text for which a hierarchical title needs to be generated and a pre-set prompt sentence;

[0169] The long text and the prompt sentence are concatenated to obtain a complete input text;

[0170] Converting the input text into input features;

[0171] Process the input features using the trained and fine-tuned large model to obtain target text with graded title information;

[0172] The target text is matched to the long text and outputted.

[0173] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0174] The communication interface is used for communication between the above terminal and other devices.

[0175] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0176] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0177] like Figure 5 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium 1401 is run on a computer, the computer executes the document hierarchical title generation method based on the large language model described in the above embodiment.

[0178] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the document hierarchical title generation method based on a large language model described in the above embodiment.

[0179] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0180] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0181] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.< / eos> < / eos> < / eos>

Claims

1. A document hierarchical title generation method based on a large language model, characterized in that: include: Receive the long text for which a hierarchical title needs to be generated and a pre-set prompt sentence; The long text and the prompt sentence are concatenated to obtain a complete input text; Converting the input text into input features; Process the input features using the trained and fine-tuned large model to obtain target text with graded title information; The target text is matched to the long text and outputted.

2. The method according to claim 1, characterized in that The long text is a document with more than 500 words and more than one paragraph; The target text includes at least two levels of titles: each level of title format is: line number, title level, title content; The input features include: input_ids array and attention_mask array.

3. The method according to claim 1, characterized in that: The long text and the prompt statement are concatenated to obtain a complete input text, including: According to the information of the splicing in the training phase, the prompt sentence is inserted into the designated position of the long text to obtain the complete input text; The prompt statement includes: generating a hierarchical title, outputting an insertion position, a title level, and a title content.

4. The method according to claim 1, characterized in that: The input features are processed using a trained and fine-tuned large model to obtain target text with hierarchical title information, including: Input the input features into the trained and fine-tuned large model, which predicts the token of the next position. Each time, the token with the highest probability at that position is taken and concatenated to the input features, and the attention_mask is modified accordingly. Repeat the above process until the predicted token is the termination symbol. <eos> , get the predicted sequence;< / eos> The prompt sentence and the original text are removed from the prediction sequence to obtain the target text.

5. The method according to claim 1, characterized in that The method further comprises: According to a pre-given training data set, a large language model is trained to obtain a large model for training and fine-tuning; The training and fine-tuning large model includes: at least two categories of sub-models; During the training process, the sub-model of each category is trained using the long document training data of that category; Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including: According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

6. The method according to claim 1, characterized in that After receiving the long text for which a hierarchical title needs to be generated and the preset prompt statement, the method includes: Using a classification model to classify the long text to obtain category information; Accordingly, the input features are processed using the trained and fine-tuned large model to obtain target text with hierarchical title information, including: According to the category information of the input feature, the input feature is processed using a sub-model corresponding to the category information to obtain a target text with hierarchical title information.

7. A document hierarchical title generation device based on a large language model, characterized in that: include, A receiving module, used for receiving a long text for which a hierarchical title needs to be generated and a preset prompt sentence; A splicing module, used for splicing the long text and the prompt sentence to obtain a complete input text; A conversion module, used for converting the input text into input features; A target text generation module, used to process the input features using the trained and fine-tuned large model to obtain a target text with hierarchical title information; An output module is used to match the target text into the long text and output it.

8. The device according to claim 7, characterized in that Also includes: A large model training module is used to train a large language model based on a pre-given training data set to obtain a large model for training and fine-tuning; The training and fine-tuning large model includes: at least two categories of sub-models; During the training process, the sub-model of each category is trained using the long document training data of that category; The target text generation module is specifically used to process the input features according to the category information of the input features using the sub-model corresponding to the category information to obtain the target text with the hierarchical title information.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 6 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 6.

Images