A document hierarchical title generation method and device based on a large language model

Abstract

Embodiments of the present application provide a document hierarchical title generation method and device based on a large language model, the method comprising: receiving a long text for which a hierarchical title needs to be generated and a pre-set prompt sentence; splicing the long text and the prompt sentence to obtain a complete input text; converting the input text into input features; processing the input features using a large model trained and fine-tuned to obtain a target text with hierarchical title information; and matching the target text to the long text and outputting. The above method solves the technical problem that authors themselves review and organize text content, or other readers search for effective information, which causes different degrees of trouble in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a document hierarchical title generation method based on a large language model, a document hierarchical title generation device based on a large language model, an electronic device and a computer readable medium. BACKGROUND

[0002] A large language model is usually based on a deep learning architecture, such as the Transformer structure leading the era in the field of natural language processing. The parameter quantity is huge, most of which has a parameter quantity of several billion, hundreds of billions or even thousands of billions. Through training on a huge amount of text corpus, it has good generalization and generalization ability, and performs excellently on downstream tasks (such as text summarization, text translation, sentiment analysis, etc.). It can support long texts on input and output, which also enables the large language model to handle various tasks, including complex learning and reasoning.

[0003] When writing long articles such as essays and blogs, the author often does not arrange the perfect logic in advance, and the important information points are distributed in the long chapter. For general readers, it is difficult to quickly grasp the key points of each module when reading a long document without hierarchical titles, and direct reading of the entire article is undoubtedly time-consuming. A long document without hierarchical titles causes different degrees of difficulty for the author to review and organize the text content, or for other readers to retrieve effective information.

[0004] Therefore, it is necessary to extract and classify the titles of such long chapter texts. SUMMARY

[0005] The embodiments of the present application provide a document hierarchical title generation method and device based on a large language model to solve the technical problem that the author reviews and organizes the text content, or other readers retrieve effective information, which causes different degrees of difficulty in the prior art.

[0006] The embodiments of the present application disclose a document hierarchical title generation method based on a large language model, comprising:

[0007] receiving a long text requiring generation of hierarchical titles and a pre-set prompt sentence;

[0008] splicing the long text and the prompt sentence to obtain a complete input text;

[0009] converting the input text into input features;

[0010] processing the input features using a large model trained and fine-tuned to obtain a target text with hierarchical title information;

[0011] Matching the target text to the long text and outputting.

[0012] Optionally, the long text is a document with more than one paragraph and greater than 500 words.

[0013] The target text includes at least two levels of titles, and each level of title is in the format of line number, title level, and title content.

[0014] The input features include an input_ids array and an attention_mask array.

[0015] Optionally, the long text and the prompt sentence are spliced to obtain a complete input text, which includes:

[0016] According to the spliced information in the training phase, the prompt sentence is inserted into a specified position of the long text to obtain a complete input text.

[0017] The prompt sentence includes generating hierarchical titles, output insertion positions, title levels, and title content.

[0018] Optionally, the input features are processed using a large model trained and fine-tuned to obtain a target text with hierarchical title information, which includes:

[0019] The input features are input into a large model trained and fine-tuned, which predicts the token at the next position. Each time, the token with the maximum probability at the position is spliced to the input features, and the attention_mask is modified accordingly. Repeat the above process until the predicted token is a terminal symbol. <eos>obtaining a predicted sequence;

[0020] obtaining a target text by removing the prompt sentence and the original text from the predicted sequence.

[0021] Optionally, the method further comprises:

[0022] training a large language model according to a pre-defined training data set to obtain a trained large model;

[0023] The trained large model comprises at least two category sub-models.

[0024] Each category sub-model is trained using long document training data of the category during the training process.

[0025] Correspondingly, processing the input feature using the trained large model to obtain the target text with hierarchical title information comprises:

[0026] processing the input feature using the sub-model corresponding to the category information of the input feature to obtain the target text with hierarchical title information.

[0027] Optionally, after receiving the long text requiring generation of hierarchical titles and the pre-set prompt sentence, the method comprises:

[0028] classifying the long text using a classification model to obtain category information;

[0029] Correspondingly, processing the input feature using the trained large model to obtain the target text with hierarchical title information comprises:

[0030] processing the input feature using the sub-model corresponding to the category information of the input feature to obtain the target text with hierarchical title information.

[0031] The embodiment also discloses a document hierarchical title generation device based on a large language model, comprising:

[0032] a receiving module configured to receive a long text requiring generation of hierarchical titles and a pre-set prompt sentence;

[0033] a splicing module configured to splice the long text and the prompt sentence to obtain a complete input text;

[0034] a conversion module configured to convert the input text into an input feature;

[0035] a target text generation module configured to process the input feature using a trained large model to obtain a target text with hierarchical title information;

[0036] An output module is configured to match the target text to the long text and output the target text.

[0037] Optionally, the document hierarchical title generation device based on the large language model further comprises:

[0038] A large model training module is configured to train a large language model according to a pre-defined training data set to obtain a training fine-tuned large model.

[0039] The training fine-tuned large model comprises at least two category sub-models.

[0040] Each category sub-model is trained using long document training data of the category during the training process.

[0041] A target text generation module is configured to process the input feature using a sub-model corresponding to category information of the input feature to obtain a target text with hierarchical title information.

[0042] The embodiment of the application further discloses an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0043] The memory is used to store a computer program.

[0044] The processor is used to execute the program stored on the memory to realize the method according to the embodiment of the application.

[0045] The embodiment of the application further discloses one or more computer readable media having instructions stored thereon, which, when executed by one or more processors, cause the processors to execute the method according to the embodiment of the application.

[0046] The embodiment of the application has the following advantages:

[0047] In the embodiment of the application, the powerful large language model technology is used to generate hierarchical titles on the basis of created content, which is more conducive to obtaining hierarchical titles with clear content division and clear structure.

[0048] Specifically, the embodiment of the application integrates the title insertion position, the title level and the title content information into a single target text segment through a specific format, uses a generative large model to perform overall prediction at one time, does not need to disassemble the task, realizes serial output one by one, realizes end-to-end result generation, and effectively improves the performance.

[0049] Meanwhile, the embodiment of the application uses a suitable prompt sentence to activate the large model pre-training stage memory, only needs a small amount of fine-tuning data to fine-tune the model, and improves the performance of the model.

[0050] Further, in the model fine-tuning evaluation stage, the title insertion position and title level output by the model are evaluated using accuracy as the evaluation index, and the title content output by the model is evaluated using BLUE as the evaluation index. By evaluating the respective evaluation indexes of the three different tasks, the performance of the fine-tuned model can be more comprehensively evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a step flowchart of a document hierarchical title generation method based on a large language model provided in an embodiment of the present application;

[0052] Figure 2 is a structural block diagram of a document hierarchical title generation device based on a large language model in an embodiment of the present application;

[0053] Figure 3 is a general flowchart provided in an embodiment of the present application;

[0054] Figure 4 is a block diagram of an electronic device provided in an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a computer readable medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0057] A large language model (LLM) is an artificial intelligence model that can give the required result in text form according to the input natural language text and prompt sentence.

[0058] The embodiment of the present application provides a document hierarchical title generation method based on a large language model. The text generation technology based on a large language model can be used to generate hierarchical titles, and the positioning of title positions and the determination of title levels can be realized at the same time. In the embodiment of the present application, the powerful large language model technology is used to generate hierarchical titles on the basis of creative content, which is more conducive to obtaining hierarchical titles with clear content division and clear structure.

[0059] The hierarchical title generation of a long document aims to generate a first-level title, a second-level title, etc. for a long document without subdivided sub-titles. Generating hierarchical titles for a long document is conducive to clarifying the structure and level of the document, combing the article logic, clearly defining the core content of each section, assisting in completing knowledge induction, and further constructing a knowledge graph.

[0060] With reference to Figure 1 and Figure 3 , a method for generating hierarchical titles based on a large language model is provided, which includes:

[0061] 101. receiving a long text that needs to generate hierarchical titles and a pre-set prompt sentence;

[0062] 102. splicing the long text and the prompt sentence to obtain a complete input text;

[0063] 103. converting the input text into input features;

[0064] 104. processing the input features using a trained and fine-tuned large model to obtain a target text with hierarchical title information;

[0065] 105. matching the target text to the long text and outputting.

[0066] The long text is a document with more than one paragraph and more than 500 words;

[0067] The target text includes at least two levels of titles: each level of title format is: line number, title level, title content;

[0068] The input features include: input_ids array, attention_mask array.

[0069] Wherein, the input_ids array is an integer array representing the tokenized version of the input text. Each integer is an index in the vocabulary, corresponding to a specific word or subword (depending on the tokenization strategy used). For example, if a Byte Pair Encoding (BPE) tokenizer is used, a word may be split into multiple subwords. Each word or subword is mapped to a unique integer identifier, and these identifiers make up the input_ids array.

[0070] Similarly, the attention_mask array is also an integer array, usually with the same length as input_ids. Its role is to tell the model which positions of the tokens are valid and which are padding. When processing text sequences of different lengths, shorter sequences are padded to a fixed length. To prevent the model from wasting attention resources on these padded positions, the attention_mask array is used to mask these positions. Usually, the positions of valid tokens are set to 1, and the positions of padding are set to 0.

[0071] In this embodiment, converting the input text into input features specifically includes:

[0072] In the BERT model and its derivatives, the input text is first processed by a tokenization procedure, in which the text is subdivided into words or subwords, each token is then mapped to a unique integer identifier, and these identifiers form the input_ids array, which represents the numerical form of the text. In order to adapt to the needs of model processing, the length of the input_ids array is normalized to a fixed value. In this normalization process, inputs longer than the predetermined value are truncated, while inputs shorter than this length are padded by adding special padding markers. This processing mechanism ensures the consistency of model input, allowing the model to batch process text data of different lengths.

[0073] For example, assume the input text is "Hello, world", which needs to be converted into a numerical representation that the model can understand. Suppose the tokenizer divides "Hello," into

[7592] and "world" into

[2088] , and the ID of the special token

CLS

[101] and the ID of the special token

SEP

[102] . Therefore, the corresponding input_ids array is: 【101, 7592, 2088, 102】. To fill it to a fixed length, assume the fixed length is 10, the corresponding input_ids array needs to be expanded to a length of 10, which is achieved by adding

PAD

[0074] And for the expanded input_ids array, the corresponding attention_mask array is: 【1,

[0075] Therefore, the input_ids array provides an efficient numerical representation of the input text, and the attention_mask array ensures that the model can effectively identify and focus on the substantive content when processing these numerical information, excluding irrelevant padding effects.

[0076] The input_ids array and the attention_mask array together form the basis of the model processing text information, which is crucial for ensuring the performance and analysis accuracy of the model.

[0077] In this embodiment, the long text and the prompt sentence are spliced to obtain a complete input text, including:

[0078] According to the spliced information in the training phase, the prompt sentence is inserted into the specified position of the long text to obtain a complete input text;

[0079] The prompt sentence is inserted into a specified position of the long text to obtain a complete input text, specifically comprising:

[0080] First, the long text is checked for format. If the long text has format problems, such as containing extra white space characters, special symbols, etc., it needs to be preprocessed. For example, text processing functions can be used to remove extra spaces, line breaks, etc., to organize the long text into a clean and regular form.

[0081] The format of the prompt sentence is also checked to ensure that there are no syntax errors or inappropriate punctuation marks. If the prompt sentence is user input, it may need to be simply cleaned up, such as removing extra spaces at the beginning and end.

[0082] If the long text is an article and you want to insert the prompt sentence after the first paragraph, you can first find the end position of the first paragraph by searching for line breaks or specific paragraph markers, and then use that position as the insertion point for the prompt sentence.

[0083] If the long text is stored in a file, it needs to be read into memory for processing. File reading operations can be used to read the long text line by line into a string or string array.

[0084] After determining the insertion point, modify the long text in memory and insert the prompt sentence. Then write the modified content back to the file. When writing back to the file, pay attention to the original format of the file, such as preserving the original line breaks and other format information.

[0085] The prompt sentence includes generating hierarchical titles, output insertion positions, title levels, and title content. The prompt sentence in this embodiment can be a pre-designed sentence and does not require user input or editing for any processing.

[0086] In the specific implementation process, the long text refers to the main text content that needs to be processed or generated, for example, it can be an article, a conversation, a story, etc.

[0087] The prompt sentence refers to additional instructions or information provided to the model to guide the model on how to process the long text.

[0088] In this embodiment, the trained and fine-tuned large model is used to process the input features to obtain target text with hierarchical title information, comprising:

[0089] The input features are input into the trained and fine-tuned large model, which predicts the token at the next position. Each time, the token with the maximum probability at that position is concatenated to the input features, and the attention mask is modified accordingly. Repeat the above process until the predicted token is the end-of-sequence token. <eos>The predicted sequence is obtained;

[0090] The target text is obtained by removing the prompts and the original text from the predicted sequence.

[0091] Specifically, this large model makes predictions based on its internal neural network architecture. Taking the Transformer architecture as an example, the multi-head attention mechanism in the model pays attention to each token in the input features from different angles, capturing the semantic relationships between them.

[0092] Then, the attention-processed information is further transformed and processed using a feedforward neural network.

[0093] When predicting the token for the next position, the large model calculates the probability distribution of each possible token appearing at the current position based on learned language patterns, semantic information, and other data. This probability distribution is based on the model's learning from a large amount of text data (the data used during training and fine-tuning). For example, for a natural language generation task, if the input feature is "today's weather," the model might calculate the probability of different tokens such as "sunny," "cloudy," and "rainy" appearing at this position based on existing knowledge.

[0094] Large models learn extensive linguistic knowledge during the pre-training phase, including word co-occurrence relationships and grammatical structures. During the fine-tuning phase, parameters are adjusted for specific tasks or domains, allowing the model to better adapt to task requirements when predicting the next token. For example, in a fine-tuned large model for news generation, when inputting features related to financial news, the model utilizes financial vocabulary and expression patterns learned during the fine-tuning phase to predict the next token.

[0095] After the large model predicts the probability distribution of the token at the next position, the token with the highest probability is selected. This selection process is based on the model's understanding of the current input features and existing knowledge. For example, if at a certain position the probability of the token "beautiful" is 0.3, the probability of "pretty" is 0.25, and the probabilities of other tokens are even lower, then "beautiful" will be selected as the token for that position.

[0096] The selected token is appended to the end of the input feature. This causes the input feature to grow continuously, gradually building a complete prediction sequence. For example, if the initial input feature is "spring", and the next selected token is "of", then the new input feature becomes "spring of".

[0097] After concatenating a new token to the input features, the attention mask needs to be modified accordingly. For example, if the original attention mask represents attention to the first three tokens of the input features, after concatenating a new token, the attention mask needs to be adjusted to correctly represent the attention to the new input features (including the newly added token). This may involve adding an element to the mask or adjusting the value of an existing element to reflect the changes in the input features.

[0098] The previous steps are repeated continuously, i.e., the large model predicts the token at the next position, selects the token with the highest probability, and concatenates the token to the input features to modify the attention mask. Each iteration increases the input features by one token, gradually building a complete prediction sequence.

[0099] In each iteration, the model is based on the current updated input features for prediction. For example, as the iteration progresses, the input features may change from "spring scenery" to "spring scenery is", then to "spring scenery is beautiful", etc. The model will make predictions for the next token based on the updated input features each time.

[0100] Continue iterating until the predicted token is the end symbol. This represents the end of a sequence, and when the model predicts this symbol, it means that the prediction sequence is complete. For example, when generating an article, when the symbol is predicted, it means that the article has been generated, and the entire sequence obtained from the initial input features through multiple token concatenations is the prediction sequence.

[0101] Further, in the prediction sequence, it is necessary to accurately identify which parts are prompt statements and original text. The prompt statement includes: generating a hierarchical title, outputting an insertion position, a title level, and title content. The original text is the original input long text. Once the prompt statement and the original text are identified, they are removed from the prediction sequence. The remaining part is the target text.

[0102] In the specific implementation process, the document hierarchical title generation method based on a large language model further comprises:

[0103] According to the pre-given training data set, the large language model is trained to obtain a trained fine-tuned large model;

[0104] The trained fine-tuned large model comprises at least two categories of sub-models.

[0105] Each category of sub-model is trained using long document training data of that category during training.

[0106] The large model is composed of at least two categories of sub-models, and the requirements of different categories of input features need to be considered in actual use. For example, for different fields (such as news, technology articles, novels, etc.), targeted sub-models are needed.

[0107] In this embodiment, the pre-defined training data set contains long documents of multiple categories and their corresponding hierarchical titles. The training data set can be divided into the following parts:

[0108] Long document: the content of each document;

[0109] Prompt statement: please generate hierarchical titles at appropriate positions for the article, and output the insertion position, title level, title content, etc.

[0110] Category information: the category to which the document belongs, such as news, technology articles, novels, etc.

[0111] Note: the correct hierarchical title and its insertion position.

[0112] Divide the training data set into different categories, and each category corresponds to a sub-model. For example, the data set can be divided into "news", "technology articles", "novels", etc.

[0113] The specific training steps are as follows:

[0114] Concatenate the long document and the prompt statement into a complete input text, separated by special markers;

[0115] Select a large language model suitable for the task, such as BERT, GPT, T5, etc.

[0116] Set training parameters such as learning rate, batch size, number of training rounds, etc.

[0117] Train the model using the training data set to ensure that the model can learn the basic ability to generate hierarchical titles.

[0118] Correspondingly, the trained large model processes the input features to obtain target text with hierarchical title information, including:

[0119] According to the category information of the input features, use the sub-model corresponding to the category information to process the input features to obtain target text with hierarchical title information.

[0120] In this embodiment, after receiving the long text and the prompt statement that need to generate hierarchical titles, it includes:

[0121] Classify the long text using a classification model to obtain category information;

[0122] The classification model in this embodiment is a Naive Bayes classification model or a convolutional neural network classification model in deep learning.

[0123] The Naive Bayes classification model is a probability classification method based on Bayes' theorem, which assumes that the features are independent of each other. This model has a wide range of applications in text classification and other fields, and can handle large amounts of text data with fast computing speed. The core of the Naive Bayes model is to calculate the posterior probability of a given data feature, and select the class with the maximum posterior probability as the prediction result.

[0124] In this embodiment, the Naive Bayes classification model is used to classify the long text to obtain the category information, which specifically includes:

[0125] Calculate the prior probability of each class, i.e. the frequency of each class appearing in the training set;

[0126] Calculate the probability of each feature under the condition of each class, which usually involves calculating the number of times the feature value appears in the sample of a particular class divided by the total number of samples in that class;

[0127] For a new sample, calculate the posterior probability of belonging to each class, i.e. the probability of each class given the sample features, and select the class with the maximum posterior probability as the prediction result.

[0128] For example, for a long text, the probability of the long text belonging to each class is calculated by counting the probability of each word in the text appearing in different classes. For example, in news classification, if the "economic" category of news often contains words such as "stock" and "exchange rate", when these words appear frequently in the long text, the probability of the long text belonging to the "economic" category will increase.

[0129] The convolutional neural network classification model in deep learning treats the long text as a one-dimensional sequence. Through the convolution operation of the convolution kernel in the convolution layer on the text sequence, local features in the text are automatically extracted, then the features are compressed through the pooling layer, and finally the classification is performed through the fully connected layer.

[0130] Correspondingly, the input features are processed using the trained and fine-tuned large model to obtain the target text with hierarchical title information, which includes:

[0131] According to the category information of the input features, the input features are processed using the sub-model corresponding to the category information to obtain the target text with hierarchical title information.

[0132] In this embodiment, the input features are processed using the sub-model corresponding to the category information to obtain the target text with hierarchical title information, which specifically includes:

[0133] Input the input feature into the submodel under the training fine-tuned large model, predict the token of the next position, take the token with the maximum probability at this position each time and splice it after the input feature, and correspondingly modify the attention_mask; repeat the above process until the predicted token is the terminal symbol <eos>obtaining a predicted sequence;

[0134] eliminating the prompt statement and the original text from the predicted sequence to obtain a target text.

[0135] In the specific implementation process, after obtaining the target text with hierarchical title information, the target text needs to be post-processed to check the accuracy, logic and coherence of the target text with hierarchical title information, to ensure that the hierarchical relationship between the hierarchical titles is correct and can accurately reflect the main content of the input features. If it is found that the target text has problems, such as unclear semantics or chaotic hierarchy, adjustment and optimization are needed.

[0136] The post-processing of the target text specifically includes:

[0137] grammar checking, semantic accuracy checking, logical relationship checking and redundant information removal.

[0138] The grammar checking is to check whether the structure of the target text is complete, for example, whether there is a lack of subject, predicate or object, to ensure that the sentence structure is complete in the formal target text with hierarchical title information.

[0139] The semantic accuracy checking needs to determine whether the concepts in the target text are consistent with the content of the input features. For example, if the input features mainly discuss the application of artificial intelligence in the medical field, and the title appears a statement about the application of artificial intelligence in the financial field, there is a problem of inconsistent concepts, and the hierarchical title information needs to be modified to ensure semantic accuracy.

[0140] Further, it is also necessary to check whether the target text completely conveys the key information in the input features. For example, the input features are about the research and development process, efficacy and side effects of a new drug, and the title only mentions the research and development process, so the title lacks semantic completeness and needs to supplement relevant information, such as "New drug: research and development process, significant efficacy and potential side effects”.

[0141] The logical relationship checking is to check the hierarchical relationship and order of the hierarchical title information. For example, the first-level title is "the growth process of plants", and the second-level titles "seed germination", "seedling growth" and "plant maturity" should all be reasonable subdivisions of the first-level title, and cannot appear logical disconnection or hierarchical confusion between the second-level title and the first-level title.

[0142] In addition, if the input features describe the development of events in chronological order, the title should also be arranged in chronological order to reflect logical coherence.

[0143] Redundancy information removal is to check whether there are repeated expressions in the target text and delete them. For example, "New material: characteristics and application of new material", where "new material" is repeated, can be modified to "New material: characteristics and application";

[0144] At the same time, remove the information in the target text that is irrelevant to the input feature content. For example, the input feature is about historical events, and modern popular culture elements appear in the hierarchical title, which is irrelevant content and needs to be removed.

[0145] The method of the embodiment provides a method for document creators to generate hierarchical titles, refine and sort article content in long documents such as essays and blogs; at the same time, it generates hierarchical titles for a long document without hierarchical titles for readers, so that readers can quickly grasp the key points of each module and save reading time; further, it divides the long document into clear and clear hierarchical titles, assists in completing knowledge induction, and facilitates further knowledge graph construction.

[0146] It should be noted that for the method embodiment, in order to simply describe, it is expressed as a series of action combinations, but those skilled in the art should know that the embodiment of the present application is not limited by the described action sequence, because according to the embodiment of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiment of the present application.

[0147] Referring to Figure 2 , a document hierarchical title generation device based on a large language model is shown, which comprises:

[0148] The receiving module is configured to receive a long text for which a hierarchical title needs to be generated and a pre-set prompt sentence;

[0149] The splicing module is configured to splice the long text and the prompt sentence to obtain a complete input text;

[0150] The conversion module is configured to convert the input text into input features;

[0151] The target text generation module is configured to process the input features using a large model trained and fine-tuned to obtain a target text with hierarchical title information;

[0152] The output module is configured to match the target text to the long text and output.

[0153] In the embodiment, the input features are input_ids array and attention_mask array.

[0154] Where input_ids is an array of integers representing the tokenized version of the input text. Each integer is an index into a vocabulary that corresponds to a particular word or subword (depending on the tokenization strategy used). For example, if a Byte Pair Encoding (BPE) based tokenizer is used, a word might be split into multiple subwords. Each word or subword is mapped to a unique integer identifier, and these identifiers make up the input_ids array.

[0155] Similarly, the attention_mask array is also an array of integers, usually with the same length as input_ids. Its purpose is to tell the model which positions of the tokens are valid and which are padding. When dealing with text sequences of different lengths, shorter sequences are padded to a fixed length. To prevent the model from wasting attention resources on these padded positions, the attention_mask array is used to mask these positions. Typically, valid token positions are set to 1, while padding positions are set to 0.

[0156] A large model training module is configured to train a large language model according to a preconfigured training data set, and obtain a training-fine-tuned large model;

[0157] The training-fine-tuned large model includes at least two category sub-models;

[0158] Each category sub-model is trained using long document training data of the category during the training process;

[0159] A target text generation module is configured to process the input feature using the sub-model corresponding to the category information of the input feature, and obtain a target text with hierarchical title information.

[0160] In this embodiment, a classification model is used to classify the long text and obtain category information;

[0161] The classification model is a Naive Bayes classification model or a convolutional neural network classification model in deep learning;

[0162] The Naive Bayes classification model is a probability classification method based on Bayes' theorem, which assumes that features are independent of each other. This model has a wide range of applications in text classification and other fields, and it can handle large amounts of text data and has fast computing speed. The core of the Naive Bayes model is to calculate the posterior probability of a given data feature, and select the class with the maximum posterior probability as the prediction result.

[0163] The convolutional neural network classification model in deep learning regards the long text as a one-dimensional sequence. The convolution kernel in the convolution layer slides on the text sequence to perform convolution operation, automatically extracts local features in the text, then performs feature compression through the pooling layer, and finally performs classification through the full connection layer.

[0164] For the device embodiment, it is basically similar to the method embodiment, so the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0165] In addition, the embodiment of the present application also provides an electronic device, such as Figure 4 As shown in the figure, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302 and the memory 1303 complete the communication among each other through the communication bus 1304,

[0166] The memory 1303 is used to store computer programs;

[0167] The processor 1301 is used to execute the programs stored in the memory 1303, and realizes the following steps:

[0168] Receiving a long text that needs to generate a hierarchical title and a pre-set prompt sentence;

[0169] Splicing the long text and the prompt sentence to obtain a complete input text;

[0170] Converting the input text into input features;

[0171] Using the trained fine-tuned large model to process the input features to obtain a target text with hierarchical title information;

[0172] Matching the target text to the long text and outputting.

[0173] The communication bus mentioned above can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0174] The communication interface is used for communication between the terminal and other devices.

[0175] The memory can include a random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0176] The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0177] As shown in Figure 5 In yet another embodiment provided by the present application, a computer readable storage medium 1401 is also provided, and the computer readable storage medium 1401 stores instructions, which, when executed on a computer, cause the computer to perform the large language model-based document hierarchical title generation method described in the above embodiments.

[0178] In yet another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when executed on a computer, causes the computer to perform the large language model-based document hierarchical title generation method described in the above embodiments.

[0179] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented in software, all or some of the steps can be implemented in the form of one or more computer programs which are stored in a computer readable medium. The computer readable medium can include one or more of a computer readable storage medium and a computer readable signal medium. The computer readable storage medium can include one or more of a magnetic storage medium, an optical storage medium, a solid state storage medium, and the like. The computer readable signal medium can include one or more of a computer readable electrical, optical, acoustical or other form of propagated signals including computer readable program code. The computer readable program code can be downloaded from one or more of a website, a server, a cloud computing system, or the like.

[0180] It should be noted that, in the present document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Also, the terms "comprising", "containing", or any other similar term are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not necessarily include only those elements in the list, but can include other elements not expressly listed, or also include elements inherent in such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0181] Each of the embodiments in the present document is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the description of the method embodiments.

[0182] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.< / eos> < / eos> < / eos>

Claims

1. A method for generating hierarchical headings in documents based on a large language model, characterized in that, include: Receive long text that requires the generation of hierarchical headings and pre-defined prompts; The long text and the prompt statement are concatenated to obtain the complete input text; Convert the input text into input features; The input features are processed using a large model that has been trained and fine-tuned to obtain target text with hierarchical title information. Match the target text to the long text and output it; The long text refers to a document with more than 500 characters and one or more paragraphs; The target text includes at least two levels of headings: each level of heading is formatted as: line number, heading level, heading content; The process of concatenating the long text and the prompt statement to obtain the complete input text includes: Based on the information concatenated during the training phase, the prompt statement is inserted into the specified position of the long text to obtain the complete input text; The prompt statements include: generating hierarchical headings, outputting the insertion position, heading level, and heading content.

2. The method according to claim 1, characterized in that, The input features include: the input_ids array and the attention_mask array.

3. The method according to claim 1, characterized in that, The input features are processed using a large model that has been trained and fine-tuned to obtain target text with hierarchical title information, including: The input features are fed into a large model for training and fine-tuning. This large model predicts the token at the next position. Each time, the token with the highest probability at that position is concatenated to the input features, and the attention_mask is modified accordingly. This process is repeated iteratively until the predicted token is the termination symbol. <eos> The predicted sequence is obtained;< / eos> The target text is obtained by removing the prompts and the original text from the predicted sequence.

4. The method according to claim 1, characterized in that, The method further includes: Train a large language model based on a pre-given training dataset to obtain a large model that has been fine-tuned. The large model for training and fine-tuning includes: sub-models of at least two categories; During the training process, each sub-model of a category is trained using long document training data of that category. Accordingly, the input features are processed using a large model that has been trained and fine-tuned to obtain target text with hierarchical title information, including: Based on the category information of the input features, the sub-model corresponding to the category information is used to process the input features to obtain the target text with hierarchical title information.

5. The method according to claim 1, characterized in that, After receiving the long text that needs to generate hierarchical headings and the pre-set prompt statements, the process includes: The long text is classified using a classification model to obtain category information; Accordingly, the input features are processed using a large model that has been trained and fine-tuned to obtain target text with hierarchical title information, including: Based on the category information of the input features, the sub-model corresponding to the category information is used to process the input features to obtain the target text with hierarchical title information.

6. A document hierarchical heading generation device based on a large language model, characterized in that, include, The receiving module is used to receive long texts that need to generate hierarchical headings and pre-set prompt statements; The splicing module is used to splice the long text and the prompt statement to obtain the complete input text; The conversion module is used to convert the input text into input features; The target text generation module is used to process the input features using a large model that has been trained and fine-tuned to obtain target text with hierarchical title information. The output module is used to match the target text into the long text and output it. The long text refers to a document with more than 500 characters and one or more paragraphs; The target text includes at least two levels of headings: each level of heading is formatted as: line number, heading level, heading content; The process of concatenating the long text and the prompt statement to obtain the complete input text includes: Based on the information concatenated during the training phase, the prompt statement is inserted into the specified position of the long text to obtain the complete input text; The prompt statements include: generating hierarchical headings, outputting the insertion position, heading level, and heading content.

7. The apparatus according to claim 6, characterized in that, Also includes: The large model training module is used to train a large language model based on a pre-given training dataset to obtain a large model that has been fine-tuned. The large model for training and fine-tuning includes: sub-models of at least two categories; During the training process, each sub-model of a category is trained using long document training data of that category. The target text generation module is specifically used to process the input features based on the category information of the input features using the sub-model corresponding to the category information, and obtain target text with hierarchical title information.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-5.

9. A computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-5.

Images