Port knowledge abstract generation method based on large model

Abstract

The invention discloses a port knowledge abstract generation method based on a large model. The method comprises the following steps: firstly, aiming at different port scenes, searching corresponding port knowledge data, inputting a large amount of port related knowledge into a GPT-4 big model for knowledge distillation, constructing a corresponding knowledge text-abstract knowledge data pair, then carrying out basic data cleaning on the data pair, and carrying out data analysis on the data pair; further cleaning the data by using an index based on the confusion degree to generate high-quality training data; continuously pre-training the model by using the high-quality training data, and finally performing further instruction fine tuning training on the pre-trained model; according to the method, port knowledge is optimized in the vertical field, news, annual reports and articles related to a port can be quickly summarized, related knowledge can be extracted, and the reading efficiency and the working efficiency of workers are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and natural language technology, and in particular to a method for generating a port knowledge summary based on a large model. Background Art

[0002] Text summarization is an important research topic in the field of natural language processing (NLP). Its goal is to generate a short and concise text version that captures and reflects the main ideas and key information of the original text. In the era of information explosion, text summarization is particularly important because it can help people quickly understand a large amount of content and save time in reading and processing information.

[0003] There are two main types of text summarization: extractive summarization and generative summarization. Extractive summarization selects key sentences or phrases from the original text to form a summary without changing the original content. Generative summarization uses advanced language models to generate new sentences that may not appear directly in the original text but can convey the same main information.

[0004] With the development of deep learning and pre-trained language models, generative text summarization technology has made significant progress. Large language models, such as GPT, BERT, and their variants, have been found to perform well in a variety of text summarization tasks. These models are pre-trained on large-scale datasets to learn comprehensive representations of language, and then fine-tuned on specific summarization tasks.

[0005] With the development of artificial intelligence applications, smart ports have become an important technology in port scenarios. Using new-generation information technology, port-related businesses and management innovations are deeply integrated to make ports more intensive, efficient, convenient, safe, and green. Since port-related reports, annual reports, and journals contain a large amount of professional information and the content is complex and diverse, this brings certain difficulties to extracting key information from this information. Therefore, using automatic generation of port information summaries to help more workers who are initially entering the port field quickly establish a port knowledge base is an urgent problem to be solved. At present, text summary generation methods mainly include two methods based on statistical models and based on embedded models. Summary generation methods based on statistical models rely on word frequency rather than syntactic or semantic understanding, so they are limited in extracting grammatical and contextual relationships. In addition, the statistical model itself does not build an "understanding" of the semantics of words, sentences or documents, but simply extracts keywords and key sentences as part of the summary. The summary generation method based on the embedded model is limited in capturing deeper text semantics due to the relatively shallow neural network. Especially compared with the Transformer-based architecture, the use of traditional large language models has limited understanding of the professional information of ports. Due to the lack of continued pre-training in the port field, the basic information of major ports and port-related professional terminology are not mastered enough, and it is impossible to accurately output a summary with port knowledge that is general, consistent, fluent, etc. Summary of the invention

[0006] The purpose of the present invention is to overcome the problems existing in the above-mentioned prior art and to provide a port knowledge summary generation method based on a large model.

[0007] In order to achieve the above object, the present invention provides a method for generating a port knowledge summary based on a large model, comprising the following steps:

[0008] S1. Based on the port scenario, collect corresponding port knowledge data, input the port knowledge into the GPT-4 large model for knowledge distillation, and guide the original text to construct the corresponding original text-abstract-keyword data pair according to the prompt words;

[0009] S2. After basic data cleaning of the original text-abstract-keyword data pair, further cleaning of the data is performed using a perplexity-based indicator to generate high-quality training data;

[0010] S3. Using the high-quality training data, the model is further pre-trained, and the trained model has port knowledge;

[0011] S4, further fine-tuning the pre-trained model;

[0012] S5. Based on the summary evaluation method of the large language model, the model after instruction fine-tuning training is tested and iterated, the causes of bad cases are analyzed, and the model is optimized in a targeted manner.

[0013] Preferably, the port knowledge is input into the GPT-4 large model for knowledge distillation, and the original text-abstract-keyword data pairs corresponding to the original text are constructed according to the prompt words:

[0014] Constructing prompt words to extract background information of the port knowledge;

[0015] A triple data set of {original text}, {abstract}, and {keywords} is generated according to the background information.

[0016] Preferably, the step S2 specifically includes the following steps:

[0017] S201, removing irrelevant or redundant information from the original text-abstract-keyword data pair, and correcting incomplete or missing information;

[0018] S202. After removing irrelevant noise data, use an open source large language model to calculate the perplexity of the data.

[0019] Preferably, the step S3 specifically includes the following steps:

[0020] S301, select the open source large language model Yi-7b as the base model;

[0021] S302: Train the model using an efficient parameter fine-tuning method.

[0022] Preferably, the step S4 specifically includes the following steps:

[0023] S401, construct prompt words and divide the original text of port knowledge into different task scenarios;

[0024] S402: Based on the above prompt word construction method, the prompt word is merged into the data pair of original text-abstract-keyword, and the model is fine-tuned.

[0025] Preferably, the step S5 specifically includes the following steps:

[0026] S501, preliminarily determining the score of the output data through an evaluation method constructed by a large language model;

[0027] S502, filtering out data with relatively low screening scores;

[0028] S503, manually label the remaining data and label the data into two categories: bad cases that need to be optimized through pre-training and bad cases that need to be optimized through instruction fine-tuning.

[0029] Preferably, the criteria for screening the bad cases that need to be optimized through pre-training and instruction fine-tuning are: if the relevant knowledge is a scenario not considered in the model pre-training data, this type of bad case continues to be pre-trained for optimization; if the answer to the summary requirement does not conform to the logic of the prompt word and the model output does not conform to the topic, this type of bad case is optimized through instruction fine-tuning.

[0030] Preferably, the open source large language model is Yi-34b.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention can quickly summarize port-related news, annual reports, and articles, extract relevant knowledge, and improve the reading and work efficiency of staff.

[0033] 2. The present invention can effectively solve the training and optimization methods based on large language models in the field of vertical summarization, and optimizes the vertical field of port knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart of a method for generating a port knowledge summary based on a large model provided by the present invention;

[0035] Figure 2 A schematic diagram of a port knowledge summary generation device based on a large model provided by the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In order to solve the problems existing in traditional text summarization methods, the present invention collects high-quality port text summary data to train the model and constructs a proprietary prompt to strengthen the port knowledge summary capability of the large language model from the pre-training stage to the instruction alignment training stage. The difference from the prior art is that the large language model is used to optimize the vertical field of port knowledge summary in the port scenario.

[0038] In the present invention, firstly, for different port scenarios, such as the construction of smart logistics in ports, intelligent management of dangerous goods in ports and other scenarios, the corresponding port knowledge data are collected. Then a large amount of port-related knowledge is input into the GPT-4 large model for knowledge distillation, and its powerful contextual learning ability is used to construct the corresponding knowledge text-summary knowledge data pair, and to construct the relevant prompt words, so that the large language model can be guided by the prompt words and summarize the relevant high-quality summaries. In addition to the data filtering method based on traditional regularization, the perplexity parameter linked to the large language model training loss is used to filter the pre-training data, and the perplexity score of the text-summary pair is calculated to optimize the data distribution. After high-quality training data is input into the open source large language model, the badcase related to the training iteration is optimized using the efficient parameter fine-tuning method (lora).

[0039] The traditional summary evaluation index based on large language model is general in field and does not have the particularity of application in port scenarios. In order to optimize the evaluation accuracy of summary related to port knowledge, this paper proposes a summary evaluation method based on large language model and constructs prompt words related to port summary for evaluation.

[0040] The present invention proposes a method for generating port knowledge summaries based on a large model, such as Figure 1 As shown, the specific steps are as follows:

[0041] S1. First, based on the port knowledge data in different scenarios, such as the construction of smart logistics in ports, intelligent management of dangerous goods in ports and other scenarios, we collect the corresponding port knowledge data. This step is to expand the port data based on vertical fields, input the port knowledge into the GPT-4 large model for knowledge distillation, and use its powerful contextual learning ability to construct relevant prompt words. The prompt words here refer to the prompt words that guide the model to output high-quality summaries, and guide the original text to construct the corresponding original text-summary-keyword data pairs according to the prompt words.

[0042] S2. Based on the data initially constructed in step S1, preliminary data cleaning is performed on the data. During the cleaning process, irrelevant or redundant information can be removed, and incomplete or missing information can be corrected. After the data is initially cleaned, the data is further cleaned using the perplexity index.

[0043] S3. Based on the cleaned data in step S2, the model is further pre-trained. The trained model has port knowledge. This step is to enable the model to have the ability to conduct preliminary analysis and master port knowledge in the vertical field of port knowledge, and to prepare for further instruction fine-tuning of the subsequent model.

[0044] S4. Based on the model preliminarily trained in step S3, the model is further fine-tuned with instructions. The trained model can generate correct port knowledge summaries step by step according to the prompt words. In essence, instruction fine-tuning is a method of fine-tuning a pre-trained large language model on a set of formatted samples expressed in natural language. In order to perform instruction fine-tuning, it is first necessary to collect or construct samples that meet the instruction format. The formatted samples here refer to the formatted data cleaned by step S2, and then the large language model is fine-tuned using these formatted data in a supervised learning manner. After instruction fine-tuning, the large language model will show excellent capabilities in the port knowledge summary task.

[0045] S5, based on the summary evaluation method of the large language model, test and iterate the model trained in step S4, analyze the causes of bad cases, and optimize the model in a targeted manner. The optimized model has stronger robustness in order to optimize the bad cases of the model. Low-quality data screened out by only basic natural language evaluation indicators such as Rouge-L may actually be caused by different summary expressions. After analyzing the causes of bad cases based on the evaluation results, the model is optimized in a targeted manner.

[0046] The present invention can quickly summarize port-related news, annual reports, and articles, extract relevant knowledge, and improve the reading and work efficiency of staff. In terms of implementation, it can be installed on a computer in the form of software, and after the model is quantified and compressed, it can also be installed on edge devices such as mobile phones, or on a server, and the summary result is returned according to the user's request interface, providing a large number of summary generation.

[0047] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the examples of the present invention is described in detail with specific embodiments below.

[0048] Assume that a specific case is as follows. In the specific implementation, a certain section of port knowledge data is as follows: A large port has introduced an automated terminal loading and unloading system, which monitors the status of goods in real time through IoT devices and optimizes the transportation route using big data analysis, effectively reducing the time that goods are detained in the port. In addition, the port logistics company uses drones to count and monitor goods, which significantly improves the efficiency of cargo management. The application of these technologies and measures not only optimizes the logistics process of the port, but also greatly improves the overall operational efficiency.

[0049] Step S1 uses the powerful contextual learning ability of the GPT-4 model to generate data for this section of port knowledge data as follows:

[0050] The construction prompt words extract the background information of the port knowledge. The construction prompt words extract the background information of this case, such as the construction of smart logistics in ports, key technologies such as IoT devices (such as sensors and RFID), automated transportation tools (such as AGV automatic guided vehicles), big data analysis platforms, 5G communication networks, implementation details, etc., to ensure that the information is complete and detailed.

[0051] Based on the extracted information, a triple data set of {original text}, {abstract}, and {keywords} is generated. This abstract is a summary and conclusion of the entire case or text, providing more concise and easy-to-understand content. The abstract may include key information, important technologies, and port information, etc.

[0052] The detailed process of step S2 is as follows:

[0053] First, the data generated in step S1 is needed. The sources of this information are wide-ranging, and can be news, annual reports, journals and other sources. In the cleaning process, irrelevant or redundant information can be removed, and incomplete or missing information can be corrected. For example, irrelevant news headlines in news titles, such as "Development of the catering industry around the port", are deleted. If the annual report contains a lot of company financial information that is not related to the port, the non-port related parts should be removed. For example, information such as "the company's overall revenue increased by 10% in 2023" will be deleted.

[0054] After removing irrelevant noise data, the perplexity of this batch of summary data is calculated using an open source large language model. Perplexity is one of the standard indicators for measuring the performance of a language model. A good language model should be able to accurately predict the probability of subsequent words given the previous text. If the probability distribution predicted by the model is close to the actual probability distribution, the perplexity will be lower, indicating that the model has a strong ability to understand and generate text. Perplexity directly reflects the probability distribution of the text generated by the model. Low perplexity means that the text generated by the model is more in line with the statistical laws of natural language, and high perplexity means that the generated text may be more rigid or not in line with natural language habits. The present invention plans to use an open source large language model such as Yi-34b to calculate the perplexity of the generated data set, and screen the data outside the abnormal distribution based on the perplexity based on the 3σ principle. For example, the remaining perplexity score interval is [a, b]. At this time, the segmented annotation method is continued, and several data are sampled every 0.1 points for data annotation. The annotation result consists of three labels: the abstract fully reflects the content of the original text, the abstract basically conforms to the original text, and the abstract does not conform to the original text. For the labeled results, the data is re-screened based on the data on both sides of the abnormal distribution and optimized to a smaller range of data [a+c,bd].

[0055] The detailed process of step S3 is as follows:

[0056] The data generated in step S2 is input into the model for training to obtain a preliminary pre-training model with port knowledge, which can be implemented through the following example.

[0057] 1. Choose an open source large language model such as Yi-7b as the base model.

[0058] 2. Use efficient parameter fine-tuning methods to train the model. The Lora method freezes most of the model parameters and only relies on the linear layer and embedding layer of the training model, which greatly reduces the amount of training parameters and trains large language models on consumer-grade graphics cards.

[0059] After a round of pre-training, the model has acquired certain vertical field knowledge. At this time, in order to make the model have stronger command following ability, the model needs to be fine-tuned.

[0060] The detailed process of step S4 is as follows:

[0061] Construct prompt words and divide the original text of port knowledge into different task scenarios, such as port annual report summary, smart port case and other scenarios.

[0062] Based on the above prompt word construction method, the prompt word is merged into the data pair of the original text, keywords, and summary, and the model is fine-tuned. A possible implementation scheme of the prompt word is as follows:

[0063] Your role is a text summarization expert. You focus on extracting key port-related information from large amounts of text and generating concise and accurate summaries:

[0064] You can summarize it from the following aspects:

[0065] Port infrastructure and equipment

[0066] Cargo handling and warehouse management

[0067] Port logistics and supply chain

[0068] Port Environment and Safety Measures

[0069] Port Economy

[0070] The generated summary requirements are as follows

[0071] Accuracy: The abstract should accurately reflect the main information and core ideas of the original text without adding personal interpretation or speculation.

[0072] Conciseness: The abstract should be as concise as possible, removing redundant information while retaining the necessary details of the original text.

[0073] Objectivity: The abstract should be objective and neutral, without any subjective evaluation or emotional coloring.

[0074] Logic: The structure of the abstract should be clear, with information organized logically and easy to understand.

[0075] Original: {article}

[0076] Keywords: {key_words}

[0077] summary:

[0078] The summary is the output part of the data. Instruction fine-tuning enables the model to generate more accurate summaries based on the specified prompt words, giving the model alignment capabilities. After instruction fine-tuning training optimization, for port text data that needs to be summarized, the model can continuously adapt and adjust the weights, and finally better adapt to the port text summary task.

[0079] After continued pre-training and instruction fine-tuning, the model now has a fairly strong ability to summarize port knowledge. In order to make the model more robust, step S5 adopts a summary evaluation method based on a large language model to test and iterate the model, which can be implemented through the following example.

[0080] First, we use the evaluation method constructed by the large language model to preliminarily determine the score of the output data. A possible implementation of the prompt word scheme is as follows:

[0081] Please evaluate the quality of the summary written for a port knowledge text. Secondly, please rate each summary from four dimensions: {accuracy}, {conciseness}, {objectivity} and {logic}. The scoring standard is 1 point (worst) to 5 points (best). Article: {Article} Summary: {Summary}.

[0082] Filter out data with relatively low screening scores, that is, data with a total score of less than 8 points.

[0083] Manually annotate the remaining data. Annotate the data into two categories: bad cases that need to be optimized through pre-training and bad cases that need to be optimized through instruction fine-tuning. The screening criteria are as follows. For example, the relevant knowledge is a scenario that is not considered in the model pre-training data. For example, the user needs to have basic knowledge in the port scenario, such as understanding the case of smart port logistics. The model must also have a certain ability to answer. This type of bad case can be optimized by continuing pre-training. When the answer to the summary requirement does not conform to the logic of the prompt word, for example, the output of the model related to smart port logistics does not conform to the topic, this type of bad case can be optimized through instruction fine-tuning.

[0084] Furthermore, the present invention also provides a port knowledge summary generation device based on a large model, such as Figure 2 As shown, including:

[0085] The data collection and optimization module is used to collect data related to port knowledge and perform data cleaning and optimization.

[0086] The port knowledge summary generation module is used to input the collected knowledge into the trained port knowledge summary generation model, and the device can output the summary information corresponding to this text.

[0087] In the device provided by the present invention, the implementation of these modules can be implemented on different devices, which can be either a server or a personal PC device. For example, the base port knowledge summary generation module can be deployed on the server and implemented through interface calls; it can also be directly implemented using program codes and deployed on the corresponding graphics card through program instructions.

[0088] Furthermore, the present invention also provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the method provided in the aforementioned embodiment is implemented. In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store program codes.

[0089] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A method for generating port knowledge summaries based on a large model, characterized in that: The following steps are involved: S1. Based on the port scenario, collect corresponding port knowledge data, input the port knowledge into the GPT-4 large model for knowledge distillation, and guide the original text to construct the corresponding original text-abstract-keyword data pair according to the prompt words; S2. After basic data cleaning of the original text-abstract-keyword data pair, further cleaning of the data is performed using a perplexity-based indicator to generate high-quality training data; S3. Using the high-quality training data, the model is further pre-trained, and the trained model has port knowledge; S4, further fine-tuning the pre-trained model; S5. Based on the summary evaluation method of the large language model, the model after instruction fine-tuning training is tested and iterated, the causes of bad cases are analyzed, and the model is optimized in a targeted manner.

2. A method for generating a port knowledge summary based on a large model according to claim 1, characterized in that: The port knowledge is input into the GPT-4 large model for knowledge distillation, and the original text-abstract-keyword data pairs are constructed according to the prompt words: Constructing prompt words to extract background information of the port knowledge; A triple data set of {original text}, {abstract}, and {keywords} is generated according to the background information.

3. A method for generating a port knowledge summary based on a large model according to claim 2, characterized in that: The step S2 specifically includes the following steps: S201, removing irrelevant or redundant information from the original text-abstract-keyword data pair, and correcting incomplete or missing information; S202. After removing irrelevant noise data, use an open source large language model to calculate the perplexity of the data.

4. A method for generating a port knowledge summary based on a large model according to claim 3, characterized in that: The step S3 specifically comprises the following steps: S301, select the open source large language model Yi-7b as the base model; S302: Train the model using an efficient parameter fine-tuning method.

5. A method for generating a port knowledge summary based on a large model according to claim 4, characterized in that: The step S4 specifically comprises the following steps: S401, construct prompt words and divide the original text of port knowledge into different task scenarios; S402: Based on the above prompt word construction method, the prompt word is merged into the data pair of original text-abstract-keyword, and the model is fine-tuned.

6. A method for generating a port knowledge summary based on a large model according to claim 5, characterized in that: The step S5 specifically comprises the following steps: S501, preliminarily determining the score of the output data through an evaluation method constructed by a large language model; S502, filtering out data with relatively low screening scores; S503, manually label the remaining data and label the data into two categories: bad cases that need to be optimized through pre-training and bad cases that need to be optimized through instruction fine-tuning.

7. A method for generating a port knowledge summary based on a large model according to claim 6, characterized in that: The criteria for screening bad cases that need to be optimized through pre-training and instruction fine-tuning are as follows: if the relevant knowledge is a scenario not considered in the model pre-training data, this type of bad case continues to be pre-trained for optimization; if the answer to the summary requirement does not conform to the logic of the prompt word and the model output does not conform to the topic, this type of bad case is optimized through instruction fine-tuning.

8. The method for generating a port knowledge summary based on a large model according to claim 3 is characterized in that: The open source large language model is Yi-34b.

Images