Railway knowledge question answering method and device based on natural language model

Abstract

The invention provides a railway knowledge question answering method and device based on a natural language model, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining a railway business question; inputting the railway business question into a pre-trained railway natural language large model to obtain an answer corresponding to the railway business question; wherein the railway natural language large model is obtained by pre-training a natural language large model based on a railway natural language sample library and performing supervised fine tuning on the pre-trained natural language large model; the natural language sample library is formed by labeling data in a railway natural language knowledge base; the railway natural language knowledge base is formed by classifying and storing railway business data. Through the method provided by the invention, the railway natural language large model with professional knowledge question-answering ability is trained and generated, and the efficiency and accuracy of railway knowledge question-answering are improved; and related problems of text application scenes in the railway field are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a railway knowledge question-answering method and device based on a natural language model. Background Art

[0002] At present, the excellent language understanding and knowledge reasoning capabilities of natural language big models represented by ChatGPT have brought artificial intelligence to a new level. China Railway has carried out in-depth research and development and application of enabling technologies and intelligent systems such as artificial intelligence, big data, and the Internet of Things. It has rich artificial intelligence application scenarios, studies key technologies of railway natural language industry big models, and lays a solid foundation for railway artificial intelligence. It will accelerate the development of railway artificial intelligence and the integration and innovation of enabling technologies, and strongly support the construction of intelligent high-speed railways.

[0003] In the prior art, knowledge base question answering uses keywords in the questions to detect knowledge stored in the knowledge base in advance, and then finds relevant answers to answer. However, traditional knowledge base question answering requires regular updates of knowledge base data, and the knowledge coverage in the knowledge base can only increase with regular updates, so the amount of data in the knowledge base is limited.

[0004] How to improve the efficiency and accuracy of railway knowledge question and answering and improve the defects of text application scenarios in the railway field are technical problems that need to be solved at present. Summary of the invention

[0005] The present invention provides a railway knowledge question-answering method and device based on a natural language model, which are used to solve the defects existing in the prior art.

[0006] The present invention provides a railway knowledge question-answering method based on a natural language model, comprising the following steps: Obtain railway business issues; Inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0007] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, before inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question, the method further includes: Acquire railway business data, classify and store the railway business data according to preset classification rules, and obtain the railway natural language knowledge base; The data in the railway natural language knowledge base is labeled according to business requirements to obtain the railway natural language sample library.

[0008] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, the training process of the railway natural language large model includes: Pre-training the natural language large model based on the railway natural language sample library to obtain a pre-trained natural language large model; The pre-trained natural language large model is fine-tuned in a supervised manner based on a preset full-parameter fine-tuning method to obtain the railway natural language large model.

[0009] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, the pre-trained natural language large model is fine-tuned in a supervised manner based on a preset full-parameter fine-tuning method to obtain the railway natural language large model, including: Identify and determine a plurality of key layers in the pre-trained natural language large model; wherein the key layers are: a first preset number of input layers and a second preset number of output layers; In each update iteration process, the model weights of the third preset number of intermediate layers are frozen, and the model weights of the multiple key layers and the model weights of the fourth preset number of intermediate layers are updated; wherein the third preset number is greater than the fourth preset number, and the number of intermediate layers is the sum of the third preset number and the fourth preset number; The railway natural language large model is determined based on the model weights after updating the preset number of iterations.

[0010] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, the railway business question is input into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question, including: Based on the railway business problem, document knowledge corresponding to the railway business problem is retrieved from the railway natural language knowledge base through generative retrieval; The document knowledge corresponding to the railway business problem is input into the pre-trained railway natural language model to obtain the answer corresponding to the railway business problem.

[0011] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, the document knowledge corresponding to the railway business question is retrieved from the railway natural language knowledge base through generative retrieval based on the railway business question, including: Converting the railway natural language knowledge base into a railway knowledge feature vector base and generating a corresponding document search index; Converting the railway business question into a plurality of search statements; wherein each of the plurality of search statements has a corresponding feature vector; Based on the multiple search statements and the feature vector corresponding to each search statement, a search is performed in the railway knowledge feature vector library through the document search index to obtain multiple document knowledge corresponding to the railway business problem.

[0012] According to a railway knowledge question-answering method based on a natural language model provided by the present invention, the railway natural language knowledge base is converted into a railway knowledge feature vector base, and a corresponding document search index is generated, including: Segmenting the documents in the railway natural language knowledge base according to a preset segmentation rule to obtain a plurality of segmented documents; The plurality of segmented documents are represented by feature vectors through a pre-trained encoding model to obtain a feature vector and a document search index corresponding to each segmented document.

[0013] The present invention also provides a railway knowledge question-answering device based on a natural language model, comprising the following modules: Acquisition module, used to acquire railway business issues; A question-answering module, used to input the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the railway knowledge question and answer method based on the natural language model as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for answering railway knowledge based on a natural language model as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned railway knowledge question-answering methods based on a natural language model.

[0017] The present invention provides a railway knowledge question-answering method and device based on a natural language model, which obtains railway business questions; inputs the railway business questions into a pre-trained railway natural language model to obtain answers corresponding to the railway business questions; wherein the railway natural language model is obtained by pre-training the natural language model based on a railway natural language sample library, and performing supervised fine-tuning on the pre-trained natural language model; the natural language sample library is formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is formed by classifying and storing railway business data. It can be seen that the present invention learns on unlabeled data through the "large-scale pre-training + fine-tuning" mode, reduces the requirements for annotated data, trains and generates a railway natural language model with professional knowledge question-answering capabilities, and improves the efficiency and accuracy of railway knowledge question-answering; and the large-scale pre-trained model can be trained on a large number of text data sets, thereby learning the ability to understand and reason about railway knowledge, realizing fast and low-cost task migration and domain migration, and well solving the related problems of text application scenarios in the railway field. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the railway knowledge question-answering method based on the natural language model provided by the present invention.

[0020] Figure 2 This is the key technology diagram of the railway natural language large model provided by the present invention.

[0021] Figure 3 It is a schematic diagram of the lora fine-tuning process provided by the present invention.

[0022] Figure 4 This is a railway natural language large model retrieval enhancement technology diagram provided by the present invention.

[0023] Figure 5 This is an application scenario diagram of the railway natural language large model provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of a railway knowledge question-answering device based on a natural language model provided by the present invention.

[0025] Figure 7It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1-Figure 7 The present invention describes a railway knowledge question-answering method and device based on a natural language model.

[0028] Figure 1 is a flow chart of the railway knowledge question-answering method based on the natural language model provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: Obtain railway business issues.

[0029] It should be noted that in the railway knowledge question and answer, you can enter a variety of questions related to railway business. These questions can cover all aspects of railways, including infrastructure, operation management, safety regulations, technical details, etc.

[0030] Step 200: input the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0031] It should be noted that the railway natural language knowledge base and railway natural language sample library are the organization, processing and annotation of railway industry data based on the sample needs of the railway natural language big model. The construction of the knowledge base and sample library will effectively support the construction and application of the railway natural language big model. Figure 2 This is the key technical diagram of the railway natural language model provided by the present invention, such as Figure 2 As shown, before step 200, the construction process of the railway natural language knowledge base and the railway natural language sample library provided by the embodiment of the present invention is first described, which specifically includes: Step S210: Acquire railway business data, classify and store the railway business data according to preset classification rules, and obtain the railway natural language knowledge base.

[0032] Specifically, in order to effectively manage and apply railway natural language data, a top-level railway data knowledge structure is designed. According to the professional classification catalog to which the data belongs, the corresponding data is classified and stored to form a railway natural language knowledge base. The text data in the railway natural language knowledge base is the original data of the large model fine-tuning annotation samples, the professional data for the large model to achieve incremental learning, and the basic knowledge base for the large model to achieve document retrieval. The classification storage and permission access strategy of the knowledge base ensure the security of the large model application knowledge.

[0033] Step S220: annotate the data in the railway natural language knowledge base according to business requirements to obtain the railway natural language sample library.

[0034] Specifically, the railway natural language sample library is based on the railway natural language big model's realization capabilities and model fine-tuning requirements, and annotates the data in the knowledge base to form a sample data set that serves the big model training. For example, to train a big model with railway professional knowledge question-and-answer capabilities, it is necessary to annotate railway knowledge questions and answer pairs; to train a big model with railway data analysis capabilities, it is necessary to annotate question and database query statement pairs, and the quality of the annotated samples also directly affects the professional capabilities of the big model.

[0035] Further, see Figure 2 , the training process of the railway natural language model is described, specifically including: Step S230: pre-training the natural language large model based on the railway natural language sample library to obtain a pre-trained natural language large model.

[0036] Specifically, the large-scale pre-trained model can be trained on a large amount of text data sets, thereby learning the ability to understand and reason about railway knowledge, achieving fast and low-cost task migration and domain migration, and well solving the relevant technical problems of text application scenarios in the railway field.

[0037] Step S240: Perform supervised fine-tuning on the pre-trained natural language large model based on a preset full-parameter fine-tuning method to obtain the railway natural language large model.

[0038] It should be noted that supervised fine-tuning (SFT) is based on a large model base and uses railway professional corpus to fine-tune the large language model in a supervised learning manner. This supervised fine-tuning method is a relatively direct and effective means that can stimulate the language model's deep understanding of railway business knowledge. First, it is necessary to collect railway professional knowledge and create instruction data. Railway professional knowledge consists of question / answer pairs. The question data is the instruction or prompt provided to the model, and the answer data is the response that the model is expected to generate based on the railway business scenario, which is usually annotated by railway professionals. Then, the large model is fine-tuned through these formatted instruction data. In order to solve the problem of too many fine-tuning parameters and ensure the fine-tuning effect, an efficient fine-tuning method needs to be adopted. At present, the mainstream fine-tuning methods include LoRA, P-tuning v2 and Freeze, among which LoRA is a more commonly used method.

[0039] The basic principle of LoRA is to freeze the pre-trained model weight parameters. When the original model parameters are frozen, additional network layers are added to the model and only the parameters of these newly added network layers are trained. In the process of fine-tuning the large language model, LoRA freezes the pre-trained model weights and injects the trainable rank decomposition matrix into each layer of the Transformer architecture. For example, for the pre-trained weight matrix , whose update can be constrained by representing the latter with a low-rank decomposition:

[0040] Moreover, the rank , at this time, the corrected forward propagation calculation formula becomes:

[0041] When fine-tuning the model, Frozen, no gradient updates, only fine-tuning parameters and , compared with the model with all parameters involved, the number of parameters fine-tuned in this step is become ,and , so the amount of fine-tuning parameters is greatly reduced.

[0042] LoRA (Low-Rank Adaptation) updates the model's weights by introducing a low-rank matrix, but this method cannot achieve the same performance level as full parameter fine-tuning in some cases. This limitation depends largely on the setting of the rank r in the low-rank matrix. Although a lower r value can lead to faster training speed and lower memory consumption, it also means a reduction in the number of parameters, which limits the model's expressiveness and makes it difficult for the model to learn more complex feature relationships, which in turn leads to insufficient generalization of the model. The end result is poor performance on some tasks and cannot achieve the effect of full fine-tuning.

[0043] In addition, the LoRA fine-tuning algorithm focuses more on the weights of the bottom and top layers of the large language model (LLM). This not only maintains the consistency of input and output, but also ensures that the bottom-level weights are responsible for basic language understanding and feature extraction, while the top-level weights are more involved in the decision-making process of specific tasks. Adjusting these layers can better control the performance of the model on specific tasks, while keeping the basic capabilities of the model unchanged, which helps maintain good generalization capabilities. This approach not only improves computational efficiency, but also enhances the generalization ability and task adaptability of the model. Figure 3 This is a schematic diagram of the lora fine-tuning process provided by the present invention. For details, please refer to Figure 3 .

[0044] The sample data in the railway field is rich and covers a wide range, which means that relying solely on the LoRA (Low-Rank Adaptation) method to fine-tune large pre-trained models may not be able to fully cover all application scenarios. Given the characteristics of the railway dataset, a full-parameter fine-tuning method may be more appropriate because it can make better use of data in these specific fields to optimize model performance. However, full-parameter fine-tuning usually requires more video memory and more powerful hardware support. For example, full-parameter training of a model with 7 billion parameters (7B) requires approximately 80GB of video memory-equivalent to the configuration of a full NVIDIA H800 graphics card. But in actual experimental environments, researchers more often use graphics cards such as RTX 3090, whose video memory capacity is far lower than this requirement. In addition, if you try to fine-tune a larger model, such as a model with 14 billion parameters (14B) or 72 billion parameters (72B), the existing graphics card resources are often difficult to meet the computing requirements required for training.

[0045] Based on this, this embodiment proposes a full parameter fine-tuning algorithm that can significantly reduce video memory. In this embodiment, the input and output are kept consistent, focusing on the weights of the bottom and top layers of the model, and fine-tuning the large model with full parameters, specifically including: Step S241, identify and determine multiple key layers in the pre-trained natural language large model; wherein the key layers are: a first preset number of input layers and a second preset number of output layers.

[0046] This embodiment takes the Qwen-7b model as an example.

[0047] Specifically, the bottom layer weights of the base model are more responsible for basic language understanding and feature extraction, while the top layer weights are more involved in the decision-making of specific tasks, so it is particularly important to focus on training the layers near the input and the layers near the output. This embodiment focuses on training the embedding layer and the last three output layers.

[0048] Step S242: During each update iteration, freeze the model weights of the third preset number of intermediate layers, and update the model weights of the multiple key layers and the model weights of the fourth preset number of intermediate layers; wherein the third preset number is greater than the fourth preset number, and the number of intermediate layers is the sum of the third preset number and the fourth preset number.

[0049] Specifically, in this embodiment, except for the embedding layer and the last three output layers, which are relatively important network layers, only 2-5 layers of the intermediate layer are updated each time. Therefore, two hyperparameters (x, y) need to be set during training. x is the number of layers to be randomly updated, ranging from 2-5 layers, and y is the iteration frequency, indicating that the parameters are updated once per batch step. By training in this way, only a few neural network layers are updated each time during forward propagation, and there is no need to introduce additional adapter structures, which greatly reduces the amount of parameters for each iteration of training and speeds up the training. The overall training process is as follows: Requires N(l) layers of neural network, T number of dataset iterations, x=i, y=j

[0050] For t(1-->T): If t%y==0: Update (N(x)) parameters

[0051] Else: Freeze N(2)

[0052] Freeze N(2)~N(l-3)

[0053] Update N(1)+N(L-3)~N(L) layers

[0054] End for

[0055] Step S243: Determine the railway natural language large model based on the model weight after updating the preset number of iterations.

[0056] Based on the above full parameter fine-tuning return, not only can the number of parameters to be learned be reduced and the training speed be accelerated, but also the video memory requirement can be further reduced.

[0057] It should be noted that Retrieval-Augmented Generation (RAG) combines retrieval technology with the railway natural language big model, allowing the big model to access and utilize external information beyond the railway sample data. RAG establishes a flexible railway knowledge base, based on the railway natural language big model capabilities, to more accurately answer railway user questions and reduce the illusions of the big model; at the same time, it meets the scenarios that require instant and accurate information, such as ticket inquiries, external news, weather conditions, etc. At the same time, the big model's ability to answer questions based on documents enhances the model's interpretability.

[0058] Figure 4 The railway natural language large model retrieval enhancement technology diagram provided by the present invention is as follows: Figure 4 As shown in the figure, RAG is mainly divided into two stages. The first is the retrieval stage, which mainly converts railway knowledge into a knowledge vector feature library, and then retrieves fragments of relevant information based on user prompts or questions; the second is the content generation stage, in which the railway natural language model comprehensively utilizes the retrieved information and the knowledge of previous fine-tuning training to synthesize answers.

[0059] Specifically, step 200 inputs the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question, including: Step 210: Based on the railway business problem, document knowledge corresponding to the railway business problem is retrieved from the railway natural language knowledge base through generative retrieval.

[0060] Specifically, step 210 includes: Step 211: convert the railway natural language knowledge base into a railway knowledge feature vector base, and generate a corresponding document search index.

[0061] Specifically, step 211 includes: Segmenting the documents in the railway natural language knowledge base according to a preset segmentation rule to obtain a plurality of segmented documents; The plurality of segmented documents are represented by feature vectors through a pre-trained encoding model to obtain a feature vector and a document search index corresponding to each segmented document.

[0062] In one embodiment, the retrieval stage mainly includes slicing the documents in the knowledge base. According to the characteristics of railway documents, the documents can be segmented by paragraphs or characters. The purpose of segmentation is to modularize railway knowledge. Then, the segmented document blocks are represented by feature vectors using a coding model to vectorize the text content, form a railway knowledge feature vector library, and generate a corresponding document search index.

[0063] Step 212: convert the railway business question into a plurality of search statements; wherein each of the plurality of search statements has a corresponding feature vector.

[0064] Step 213: Based on the multiple search statements and the feature vector corresponding to each search statement, search the railway knowledge feature vector library through the document search index to obtain multiple document knowledge corresponding to the railway business problem.

[0065] In one embodiment, the user question is converted into multiple search statements and corresponding feature vectors, and matching documents are retrieved in the knowledge base, and top K documents are recommended.

[0066] Step 220: input the document knowledge corresponding to the railway business problem into the pre-trained railway natural language model to obtain the answer corresponding to the railway business problem.

[0067] In one embodiment, the top K documents are input into a railway natural language model to generate generative answers for the corresponding documents.

[0068] The railway natural language big model provided by the embodiment of the present invention is based on the natural language big model base. By constructing a railway knowledge base and sample library, an improved supervised fine-tuning (SFT) training method is used to enable the natural language big model to have railway business professional knowledge, so as to achieve the purpose of ensuring railway business data security and reducing model hallucinations. Based on the trained big model, the generative retrieval (RAG) technology is used to improve the accuracy and reliability of the model.

[0069] The above is a step description of the railway knowledge question-answering method based on the natural language model provided by the present invention. From the description of the above steps, it can be seen that according to the railway knowledge question-answering method based on the natural language model provided by the present invention, by obtaining railway business problems; inputting the railway business problems into the pre-trained railway natural language model, the answer corresponding to the railway business problem is obtained; wherein the railway natural language model is: pre-training the natural language model based on the railway natural language sample library, and the pre-trained natural language model is obtained by supervised fine-tuning; the natural language sample library is: annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: classifying and storing the railway business data. It can be seen that the present invention learns on unlabeled data through the "large-scale pre-training + fine-tuning" mode, reduces the requirements for annotated data, trains and generates a railway natural language model with professional knowledge question-answering capabilities, and improves the efficiency and accuracy of railway knowledge question-answering; and the large-scale pre-trained model can be trained on a large number of text data sets, so as to learn the ability to understand and reason about railway knowledge, realize fast and low-cost task migration and domain migration, and well solve the relevant problems of text application scenarios in the railway field.

[0070] Based on the above embodiments, in this embodiment, the training results based on the railway natural language big model are analyzed and explained. The big model for railway professional knowledge question and answer is a big model with intelligent question and answer capabilities trained using railway professional data, which mainly includes question and answer sample annotation, big model training, and model result analysis process.

[0071] 1. Sample annotations of railway knowledge questions and answers.

[0072] This example collects 60 railway textbooks, as well as some information regulations, vehicle management, electrical management, engineering management, and mechanical management regulations as sample annotation data. According to the requirements of the big model intelligent question and answer annotation, the railway data is constructed into "question-answer" pairs, that is, according to the railway knowledge description, questions are asked about knowledge points from different dimensions, and the questions are answered at the same time. A total of 100,000 sample data are annotated, and a 650-question test set is constructed for testing and evaluating the big model.

[0073] 2. Training of large model of railway knowledge question and answer.

[0074] The intelligent question-answering big model training uses 155,400 labeled "question-answer" pair sample data, and uses three different-scale big models of Tongyi Qianwen Qwen_7b_chat, Qwen_7b_int4, and Qwen_14b_int4 as the basic model to train the big model. Each model uses 2 to 4 Rtx3090 graphics cards, and it takes about 151h to 200h to train 40 epochs. The method used in this embodiment only takes 100h to train 40 epochs, which proves the effectiveness of the algorithm in this embodiment. The test results are shown in Table 1.

[0075] Table 1

[0076] 3. Analysis of large model training results.

[0077] According to the training results of the above three different models, 650 test data were used to test the model's ability. The manual evaluation method was used, that is, the manual evaluation of whether the answer generated by the model was correct. Correct means that the model answer is consistent with the standard answer, and incorrect means. The test results are shown in Table 2.

[0078] Table 2

[0079] It can be seen from the model fine-tuning training results that compared with the Qwen-7b-chat general model, the large model trained with railway corpus has a significant improvement in the accuracy of answering railway professional knowledge, and the answer accuracy has increased by at least 30 percentage points, and the Qwen-7b model has the highest accuracy. Based on the fine-tuned Qwen-7b, the STF+RAG method is adopted, that is, on the basis of the large model of fine-tuning training, a RAG retrieval method is established, and the same test data is used to test Qwen-7b-STF+ARG, with an accuracy of 85.1%. Compared with Qwen-7b-STF, the accuracy is increased by 11.3%, proving that SFT+RAG is an effective method for constructing a large railway natural language model. In addition, the recognition accuracy of the fine-tuning algorithm + RAG of this embodiment is further increased by about 7%, further verifying the effectiveness of the algorithm proposed in this embodiment. From the analysis of video memory usage, training the 7b model and fine-tuning all parameters requires about 80Gb of video memory, while the algorithm proposed in this embodiment only requires about 30Gb, which verifies that the algorithm proposed in this embodiment can further reduce hardware requirements. The results are shown in Table 3.

[0080] Table 3

[0081] Based on the above embodiments, in this embodiment, the process of building a railway artificial intelligence platform based on a natural language big model is described. The railway natural language big model is a complete artificial intelligence model. To obtain a railway professional natural language big model, the railway artificial intelligence platform needs to provide data annotation, model development, model training, model reasoning and other artificial intelligence core functions and support big model component development and application. The evolution of the railway natural language industry big model is divided into three levels: L0, L1, and L2. Therefore, the architecture design of the railway natural language big model is an evolutionary design of L0-L1-L2 based on the railway artificial intelligence platform. Figure 5 This is the railway natural language large model application scenario diagram provided by the present invention, combined with Figure 5 The whole process is explained.

[0082] 1. Large model infrastructure.

[0083] The hardware facilities of large models are intelligent computing server clusters built based on GPUs, AI acceleration cards, etc. The hardware infrastructure mainly includes hardware equipment such as artificial intelligence computing power, data storage, and networks. Due to large-scale data parallel computing during large model training, its artificial intelligence computing power cards, communication networks, and data storage must all have the characteristics of high computing power and high energy efficiency.

[0084] 2. Railway artificial intelligence platform and large model support components.

[0085] The railway artificial intelligence platform mainly provides full life cycle management for the construction of large models, including data labeling, model development, model evaluation, model reasoning, and service operation and maintenance. In order to meet the large-scale computing power scheduling of natural language large models, customized development of large models, and large-scale data processing, the platform provides large model support components. In terms of large model training, it presets large model fine-tuning algorithms, provides distributed training parallel optimization and data acceleration functions, and supports multi-source heterogeneous large model fusion architecture. In terms of large model reasoning, it provides functions such as reasoning acceleration, distributed deployment, and compression pruning to enable large models to provide efficient and stable reasoning services in a distributed environment. In terms of large model applications, it provides functions such as application development workflow and prompter engineering to achieve online fine-tuning of large models, quickly form railway professional reasoning services, and provide large model services for railway business systems.

[0086] 3. L0 natural language general large model.

[0087] L0 refers to general large models, including open source basic large models and commercial basic large models. Open source large models include the foreign LLaMa series, the domestic GLM series, Alibaba's Qwen series, etc., and provide models with different parameter levels. Generally, the larger the model scale, the stronger its generalization ability. The basic large model has strong generalization ability, but lacks industry depth, and it is difficult to provide high-value services in segmented application scenarios in the railway field.

[0088] 4. L1 Railway natural language industry big model.

[0089] L1 refers to the railway industry natural language big model, which is based on the L0 level basic big model, inputs massive railway data, and learns the patterns and rules of railway data through training to form a big model with railway knowledge. Compared with the general big model, the railway industry natural language big model can more directly penetrate into specific railway business scenarios, more accurately meet the specific needs of the railway industry, and make up for the shortcomings of the general big model that cannot be optimally adapted to vertical industries.

[0090] 5. L2 railway natural language scenario model.

[0091] L2 refers to the reasoning model for more detailed scenarios. Based on the railway industry model L1, through fine-tuning or prompt words, multiple railway business scenario application models L2 can be derived to form an accurate railway artificial intelligence application model, which can quickly empower various railway artificial application scenarios. For example, based on the railway industry natural language model, according to the application scenario requirements such as railway safety supervision and management, dispatching command parsing, etc., the L1 model is fine-tuned to achieve accurate accident fault analysis, dispatching command information extraction and other empowerment models.

[0092] The embodiment of the present invention proposes an application scenario of a railway natural language big model according to the general capabilities and characteristics of the natural language big model and the application requirements of railway artificial intelligence, designs a natural language big model architecture based on the railway artificial intelligence platform, proposes an evolution mode of the railway natural language big model L0-L1-L2, studies key natural language technologies for the application scenarios of the railway artificial intelligence big model, and especially proposes a new fine-tuning technology improvement method to carry out research and development experiments on railway text data big models, and preliminarily realizes a railway industry natural language big model with question-answering and auxiliary writing capabilities.

[0093] The railway knowledge question and answer device based on a natural language model provided by the present invention is described below. The railway knowledge question and answer device based on a natural language model described below and the railway knowledge question and answer method based on a natural language model described above can refer to each other.

[0094] Figure 6 is a schematic diagram of the structure of the railway knowledge question-answering device based on the natural language model provided by the present invention, such as Figure 6As shown, the railway knowledge question-answering device based on the natural language model provided by the present invention includes: An acquisition module 601 is used to acquire railway business issues; Question and answer module 602, used for inputting the railway business question into the pre-trained railway natural language model to obtain the answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0095] The railway knowledge question-and-answer device based on the natural language model provided by the present invention obtains railway business questions; inputs the railway business questions into the pre-trained railway natural language model to obtain the answers corresponding to the railway business questions; wherein the railway natural language model is obtained by pre-training the natural language model based on the railway natural language sample library, and performing supervised fine-tuning on the pre-trained natural language model; the natural language sample library is formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is formed by classifying and storing the railway business data. It can be seen that the present invention learns on unlabeled data through the "large-scale pre-training + fine-tuning" mode, reduces the requirements for annotated data, trains and generates a railway natural language model with professional knowledge question-and-answer capabilities, and improves the efficiency and accuracy of railway knowledge question-and-answer; and the large-scale pre-trained model can be trained on a large number of text data sets, thereby learning the ability to understand and reason about railway knowledge, realizing fast and low-cost task migration and domain migration, and well solving the related problems of text application scenarios in the railway field.

[0096] Based on the above embodiment, in this embodiment, the device further includes a library generation module, which is specifically used to: Before inputting the railway business question into the pre-trained railway natural language model to obtain the answer corresponding to the railway business question, railway business data is obtained, and the railway business data is classified and stored according to preset classification rules to obtain the railway natural language knowledge base; The data in the railway natural language knowledge base is labeled according to business requirements to obtain the railway natural language sample library.

[0097] Based on the above embodiment, in this embodiment, the device further includes a training module, which is specifically used for: Pre-training the natural language large model based on the railway natural language sample library to obtain a pre-trained natural language large model; The pre-trained natural language large model is fine-tuned in a supervised manner based on a preset full-parameter fine-tuning method to obtain the railway natural language large model.

[0098] Based on the above embodiment, in this embodiment, the device further includes a training module, which is specifically used for: Identify and determine a plurality of key layers in the pre-trained natural language large model; wherein the key layers are: a first preset number of input layers and a second preset number of output layers; In each update iteration process, the model weights of the third preset number of intermediate layers are frozen, and the model weights of the multiple key layers and the model weights of the fourth preset number of intermediate layers are updated; wherein the third preset number is greater than the fourth preset number, and the number of intermediate layers is the sum of the third preset number and the fourth preset number; The railway natural language large model is determined based on the model weights after updating the preset number of iterations.

[0099] Based on the above embodiment, in this embodiment, the question and answer module 602 is specifically used for: Based on the railway business problem, document knowledge corresponding to the railway business problem is retrieved from the railway natural language knowledge base through generative retrieval; The document knowledge corresponding to the railway business problem is input into the pre-trained railway natural language model to obtain the answer corresponding to the railway business problem.

[0100] Based on the above embodiment, in this embodiment, the device further includes a retrieval module, which is specifically used to: Converting the railway natural language knowledge base into a railway knowledge feature vector base and generating a corresponding document search index; Converting the railway business question into a plurality of search statements; wherein each of the plurality of search statements has a corresponding feature vector; Based on the multiple search statements and the feature vector corresponding to each search statement, a search is performed in the railway knowledge feature vector library through the document search index to obtain multiple document knowledge corresponding to the railway business problem.

[0101] Based on the above embodiment, in this embodiment, the retrieval module is specifically used for: Segmenting the documents in the railway natural language knowledge base according to a preset segmentation rule to obtain a plurality of segmented documents; The plurality of segmented documents are represented by feature vectors through a pre-trained encoding model to obtain a feature vector and a document search index corresponding to each segmented document.

[0102] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the railway knowledge question-answering method based on the natural language model, and the method includes: Obtain railway business issues; Inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0103] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the railway knowledge question answering method based on the natural language model provided by the above methods, the method comprising: Obtain railway business issues; Inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0105] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the railway knowledge question-answering method based on a natural language model provided by the above methods, the method comprising: Obtain railway business issues; Inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; Among them, the railway natural language big model is: pre-training the natural language big model based on the railway natural language sample library, and supervised fine-tuning the pre-trained natural language big model; the natural language sample library is: formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is: formed by classifying and storing railway business data.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A railway knowledge question-answering method based on a natural language model, characterized in that: include: Obtain railway business issues; Inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; The railway natural language big model is obtained by pre-training the natural language big model based on the railway natural language sample library and performing supervised fine-tuning on the pre-trained natural language big model; The natural language sample library is formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is formed by classifying and storing the railway business data.

2. The railway knowledge question-answering method based on a natural language model according to claim 1 is characterized in that: Before inputting the railway business question into the pre-trained railway natural language model to obtain an answer corresponding to the railway business question, the method further includes: Acquire railway business data, classify and store the railway business data according to preset classification rules, and obtain the railway natural language knowledge base; The data in the railway natural language knowledge base is labeled according to business requirements to obtain the railway natural language sample library.

3. The railway knowledge question-answering method based on a natural language model according to claim 2 is characterized in that: The training process of the railway natural language large model includes: Pre-training the natural language large model based on the railway natural language sample library to obtain a pre-trained natural language large model; The pre-trained natural language large model is fine-tuned in a supervised manner based on a preset full-parameter fine-tuning method to obtain the railway natural language large model.

4. The railway knowledge question-answering method based on a natural language model according to claim 3 is characterized in that: The method of fine-tuning the pre-trained natural language model in a supervised manner based on the preset full-parameter fine-tuning method to obtain the railway natural language model includes: Identify and determine a plurality of key layers in the pre-trained natural language large model; wherein the key layers are: a first preset number of input layers and a second preset number of output layers; In each update iteration process, the model weights of the third preset number of intermediate layers are frozen, and the model weights of the multiple key layers and the model weights of the fourth preset number of intermediate layers are updated; wherein the third preset number is greater than the fourth preset number, and the number of intermediate layers is the sum of the third preset number and the fourth preset number; The railway natural language large model is determined based on the model weights after updating the preset number of iterations.

5. The railway knowledge question-answering method based on a natural language model according to claim 1, characterized in that: The step of inputting the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question includes: Based on the railway business problem, document knowledge corresponding to the railway business problem is retrieved from the railway natural language knowledge base through generative retrieval; The document knowledge corresponding to the railway business problem is input into the pre-trained railway natural language model to obtain the answer corresponding to the railway business problem.

6. The railway knowledge question-answering method based on a natural language model according to claim 5 is characterized in that: The step of retrieving document knowledge corresponding to the railway business problem from the railway natural language knowledge base through generative retrieval based on the railway business problem includes: Converting the railway natural language knowledge base into a railway knowledge feature vector base and generating a corresponding document search index; Converting the railway business question into a plurality of search statements; wherein each of the plurality of search statements has a corresponding feature vector; Based on the multiple search statements and the feature vector corresponding to each search statement, a search is performed in the railway knowledge feature vector library through the document search index to obtain multiple document knowledge corresponding to the railway business problem.

7. The railway knowledge question-answering method based on a natural language model according to claim 6 is characterized in that: The converting the railway natural language knowledge base into a railway knowledge feature vector base and generating a corresponding document search index includes: Segmenting the documents in the railway natural language knowledge base according to a preset segmentation rule to obtain a plurality of segmented documents; The plurality of segmented documents are represented by feature vectors through a pre-trained encoding model to obtain a feature vector and a document search index corresponding to each segmented document.

8. A railway knowledge question-answering device based on a natural language model, characterized in that: include: Acquisition module, used to acquire railway business issues; A question-answering module is used to input the railway business question into a pre-trained railway natural language model to obtain an answer corresponding to the railway business question; The railway natural language big model is obtained by pre-training the natural language big model based on the railway natural language sample library and performing supervised fine-tuning on the pre-trained natural language big model; The natural language sample library is formed by annotating the data in the railway natural language knowledge base; the railway natural language knowledge base is formed by classifying and storing the railway business data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the railway knowledge question and answer method based on the natural language model as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the railway knowledge question and answer method based on a natural language model as described in any one of claims 1 to 7 is implemented.

Images