Time series prediction method of scientific and technological events based on RAG

Abstract

The present application provides a method for predicting the time series of scientific and technological events based on RAG, which includes the following steps: establishing and training an event prediction model; establishing and training a RAG retrieval model, and using the retrieval model to vectorize and store the retrieved documents; obtaining a query question, which is a question related to the prediction of scientific and technological events; the trained event prediction model generates a prediction answer based on the query question; the prediction answer includes a judgment label, and the judgment label is used to determine whether it is necessary to retrieve a first historical event to generate a description of the first historical event; if no retrieval is required, the event prediction model directly outputs the prediction answer; if retrieval is required, the retrieval model is used to retrieve the first historical event to obtain relevant scientific and technological documents, and a description of the first historical event is obtained based on the relevant scientific and technological documents, and the description of the first historical event is added to the historical basis, and the event prediction model outputs the prediction answer with the added historical basis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for predicting the time series of scientific and technological events based on RAG. Background Art

[0002] Time series forecasting is an analytical technique for predicting future trends or values ​​based on time series data. It is widely used in fields such as weather forecasting and financial forecasting. By analyzing the changing patterns of historical data in the time dimension, such as trends, seasonality, periodicity, and randomness, mathematical models and statistical methods are used to estimate future data points.

[0003] However, when it comes to predicting scientific and technological events and scientific research trends, there are problems such as complex knowledge relationships, extensive interdisciplinary cross-disciplinary studies, and a long time span. Therefore, it is difficult to accurately predict scientific and technological events and scientific research trends. Summary of the Invention

[0004] One of the purposes of this application is to provide a RAG-based time series prediction method for scientific and technological events, which can improve the accuracy of scientific and technological event predictions and scientific research trend predictions.

[0005] To achieve the above-mentioned and other related purposes, the present application provides a method for predicting the time series of scientific and technological events based on RAG, comprising the following steps:

[0006] Establish an event prediction model and train the event prediction model;

[0007] Establish a RAG retrieval model and train the RAG retrieval model;

[0008] Obtaining query questions, where the query questions are questions related to scientific and technological event predictions;

[0009] The trained event prediction model generates a predicted answer based on the query question; the predicted answer includes a judgment label, and the judgment label is used to determine whether it is necessary to use the RAG retrieval model to retrieve the first historical event to generate a description of the first historical event;

[0010] If retrieval is not required, the event prediction model directly outputs the predicted answer;

[0011] If retrieval is required, the RAG retrieval model is used to search the first historical event to obtain relevant scientific and technological documents, and a description of the first historical event is obtained based on the relevant scientific and technological documents. The description of the first historical event is added to the historical basis, and the event prediction model outputs a prediction answer with added historical basis.

[0012] This application has at least the following beneficial effects:

[0013] This application provides a RAG-based time series prediction method for scientific and technological events, which can improve the accuracy of scientific and technological event predictions and scientific research trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 Schematic diagram of the flow of a method for predicting the timing of scientific and technological events based on RAG according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following describes the embodiments of the present application through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in the present application can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0017] A scientific and technological event refers to a technological advancement or breakthrough in the field of science and technology.

[0018] Scientific and technological event forecasting refers to the prediction of technological advancements or breakthroughs that may occur in a specific scientific and technological field over a period of time. For example, rocket technology is a comprehensive technological system encompassing multiple sub-fields, primarily including rocket-related materials technology, fuel technology, and aerodynamics technology. Therefore, scientific and technological event forecasting can use the historical evidence of these three sub-fields to predict the specific content of future technological advancements or breakthroughs in rocket technology.

[0019] The present application provides a method for predicting the time series of scientific and technological events based on RAG, constructs an event prediction model based on a large language model (LLM), generates a prediction answer based on a query question, and combines retrieval-augmented generation (RAG) technology to obtain historical evidence to support the prediction answer, thereby obtaining a prediction answer with historical basis.

[0020] The query question is about the possible technological advancements or breakthroughs that may occur in a certain scientific and technological field in the future. The predicted answer is a specific prediction of the technological advancements or breakthroughs that may occur in that field in the future. For example, if the scientific and technological field is rocket technology, the predicted answer would combine historical evidence from materials technology, fuel technology, and aerodynamics technology to predict the specific advancements or breakthroughs in rocket technology.

[0021] The historical evidence should include at least one past scientific and technological event (hereinafter referred to as a historical event) and a description of the historical event. The description of the historical event should include a brief introduction to the historical event.

[0022] When there are multiple historical events, these events can be arranged in the order in which they occurred. That is, these historical events are chronological events. The historical events arranged in the order in which they occurred constitute a historical context, which includes the first historical event, the second historical event, and the third historical event. The second historical event occurred before the first historical event, and the third historical event occurred before the second historical event. Correspondingly, the historical basis includes the historical context and descriptions of the historical events in the historical context, that is, the historical basis includes the first historical event, the second historical event, and descriptions corresponding to the first and second historical events.

[0023] Reference Figure 1 The present application provides a method for predicting the time series of scientific and technological events based on RAG, comprising the following specific steps:

[0024] Establish an event prediction model and train the event prediction model;

[0025] Establish a RAG retrieval model and train the RAG retrieval model;

[0026] Obtaining query questions, where the query questions are questions related to scientific and technological event predictions;

[0027] The trained event prediction model generates a predicted answer based on the query question; the predicted answer includes a judgment label, and the judgment label is used to determine whether it is necessary to use the RAG retrieval model to retrieve the first historical event to generate a description of the first historical event.

[0028] If retrieval is not required, the event prediction model directly outputs the predicted answer.

[0029] If retrieval is required, the RAG retrieval model is used to retrieve relevant scientific and technological documents for the first historical event. A description of the first historical event is obtained based on the relevant scientific and technological documents. The description of the first historical event is added to the historical evidence, and the event prediction model outputs a predicted answer with the added historical evidence. In this case, the predicted answer includes the historical evidence with the added description of the first historical event. In some embodiments, if further retrieval is required, a corresponding historical retrieval tag is generated.

[0030] The event prediction model can directly output prediction answers by: predicting the technological progress or breakthrough that may occur in a certain scientific and technological field in the future through the evolution trend of time series events; or directly outputting prediction answers by calculating the probability of technological progress or breakthrough that may occur in a certain scientific and technological field in the future. The probability of technological progress or breakthrough that may occur in a certain scientific and technological field in the future includes calculating the probability of events that will affect the possible technological progress or breakthrough.

[0031] In some embodiments, if the historical evidence for the answer is complete and does not need to be retrieved, the corresponding generated judgment tag is a stop tag.

[0032] In some embodiments, if the prediction answer related to the scientific and technological event prediction is determined based on the development status of multiple technical fields, the judgment tag includes multiple judgment sub-tags corresponding to the multiple technical fields.

[0033] The judgment tag includes the first historical event and a preset time range, which can be the past two years, the past three years, or the past five years.

[0034] The following description is made by taking the retrieval of the first historical event as an example.

[0035] Establish a knowledge base to store scientific and technological documents. Scientific and technological documents come from science and technology-related websites, journals and books.

[0036] Retrieving the first historical event includes the following steps: using the trained RAG retrieval model to retrieve scientific and technological documents related to the first historical event within a preset time range from the knowledge base, inputting the first historical event and the retrieved scientific and technological documents related to the first historical event into the event prediction model, and generating a description of the first historical event.

[0037] The RAG retrieval model is a natural language processing model that combines retrieval and generation technologies. The RAG retrieval model can vectorize the retrieved documents and store the vectorized documents.

[0038] The trained RAG retrieval model has a single-tower structure. It works by first invoking the model offline to construct a document representation, then invoking it online to construct a query representation, and finally linking these representations through similarity calculations. Queries here include queries for the first historical event and a preset time range. Documents here represent scientific documents in the knowledge base.

[0039] Retrieving scientific and technological documents related to a first historical event within a preset time range from a knowledge base using a trained RAG retrieval model includes: constructing a query vector based on the first historical event and the preset time range; constructing multiple document vectors based on the scientific and technological documents in the knowledge base; calculating the similarity between the query vector and the multiple document vectors; obtaining a document vector with high similarity to the query vector based on the similarity between the query vector and the multiple document vectors, and obtaining the scientific and technological documents corresponding to the document vector with high similarity to the query vector.

[0040] Embedding models can be used to construct query vectors and multiple document vectors. Embedding models are one of the core technologies in natural language processing (NLP). They can convert historical events, preset time ranges, and the content of scientific documents into fixed-dimensional vector representations (also called embedding vectors). The embedding vectors here correspond to the query vectors and document vectors mentioned above.

[0041] When multiple scientific documents corresponding to document vectors with high similarity to the query vector are obtained, scientific documents with high relevance are selected and input into the event prediction model to generate a description of the first historical event. The relevance includes the relevance between the scientific document and the first historical event, and the timeliness of the scientific document and the preset time range. Specifically, a classification model trained based on the Transformers architecture can be used to perform relevance scoring, and scientific documents with high relevance can be selected based on the scoring ranking. In some embodiments, the relevance score includes two parts: the semantic relevance between the scientific document and the first historical event, and the timeliness of the scientific document and the preset time range.

[0042] The classification model trained on the Transformers architecture is a type of deep learning model. Its core is the self-attention mechanism, which effectively captures the semantic relevance and temporal characteristics of text. Semantic relevance refers to the semantic relevance of a scientific document to the primary historical event, while temporal characteristics refer to the timeliness of a scientific document within a pre-set timeframe, i.e., the time within which the document was published.

[0043] Retrieving the first historical event also includes: judging the description of the first historical event, confirming whether it is necessary to retrieve historical events that occurred before the first historical event,

[0044] If retrieval is required, a description of the historical event that occurred before the first historical event is generated, and the description of the historical event that occurred before the first historical event is added to the historical basis, and the event prediction model outputs a prediction answer with the added historical basis. At this time, the prediction answer includes the historical basis with the added description of the first historical event and the description of the historical event that occurred before the first historical event.

[0045] The historical evidence with the description of the first historical event is input into the event prediction model. If the event prediction model determines that the historical evidence with the description of the first historical event is incomplete, it will repeat the above search process to continue supplementing the relevant historical time evidence until the historical evidence is complete and the event prediction model outputs a final answer. In other words, the above judgment and search process can be repeated multiple times until the historical evidence with the description of the historical event meets the preset historical evidence requirements. The preset historical evidence requirement can be that the historical evidence is complete. Correspondingly, the predicted answer in this case includes the historical evidence with the description of the first historical event and descriptions of multiple historical events that occurred before the first historical event (including the second historical event, the third historical event, etc.). If there is no need to search for historical events that occurred before the first historical event, the search for the first historical event is completed. The historical event that occurred before the first historical event is, for example, the second historical event. The following uses the second historical event as an example to illustrate how to search for historical events that occurred before the first historical event. Retrieving the first historical event also includes the following steps: using the trained RAG retrieval model to retrieve scientific and technological documents related to the second historical event from the knowledge base, inputting the second historical event and the retrieved scientific and technological documents related to the second historical event into the event prediction model, generating a description of the second historical event, adding the description of the second historical event to the historical evidence, and having the event prediction model output a prediction answer with the added historical evidence. The prediction answer then includes the historical evidence with the description of the first historical event and the description of the second historical event added. The above judgment and retrieval process is repeated until the preset historical evidence requirements are met. The requirements include: after generating the description of the second historical event, judging the description of the second historical event to confirm whether it is necessary to retrieve a historical event that occurred before the second historical event (such as a third historical event). If retrieval is still required, then retrieving the third historical event. The retrieval process for the third historical event is similar to the retrieval process for the first and second historical events described above, and will not be repeated here.

[0046] The following is a detailed explanation of the training of the event prediction model.

[0047] Training the event prediction model includes training the predicted answers and training the accuracy of the judgment results. The judgment results include a determination of whether to search for the first historical event and a determination of whether to search for historical events that occurred before the first historical event based on the description of the first historical event.

[0048] Imitation learning can be used to train the predicted answers and the accuracy of the judgment results. Imitation learning is an algorithm that learns strategies by imitating expert examples. Expert examples include corresponding training sets, which include scientific documents, historical questions, and answers based on scientific documents and historical questions. Historical questions are historical query questions, which are questions about possible technological advances or breakthroughs in a certain scientific and technological field in the future. Answers to historical questions are answers to historical query questions, including the judgment result of whether it is necessary to search for a first historical event, and the specific content prediction corresponding to the technological advances or breakthroughs that may occur in the scientific and technological field in the future. It also includes the judgment result of judging the description of the first historical event and confirming whether it is necessary to search for historical events that occurred before the first historical event.

[0049] Imitation learning includes a loss function, which is used to measure the difference between the model's predictions and the true results.

[0050] Loss Function , activation function and the similarity of text The relevant calculation formula is as follows:

[0051] ;

[0052] To calculate the current subproblem The current total question is the query question, that is, the question of possible technological progress or breakthroughs in a certain scientific and technological field in the future. Taking rocket technology as an example, the current total question is the question of possible technological progress or breakthroughs in the field of rocket technology in the future. It is the historical basis of the first historical event related to the overall problem (such as previous breakthroughs in material technology, fuel technology or aerodynamic technology). If i>1, the question may be about the historical basis of the second or third historical event.

[0053] To calculate the probability of answering the current sub-question. Indicates the answer generated for the current sub-question, including generating a description of the first historical event and making a judgment to confirm whether it is necessary to retrieve historical events that occurred before the first historical event. Relevant scientific and technological documents can be retrieved for the first historical event. By calculating the probability of answering the current sub-question, a judgment result can be obtained on whether it is necessary to search for the first historical event (such as breakthroughs in materials technology, fuel technology, or aerodynamics technology), as well as a specific prediction of the technological progress or breakthroughs that may occur in this scientific and technological field in the future. The description of the first historical event is also judged to confirm whether it is necessary to search for historical events that occurred before the first historical event.

[0054] is to calculate the similarity of texts. The relevant scientific and technological documents that can be retrieved for the second historical event. The text content used to avoid retrieving relevant scientific and technological documents for the first historical event is similar to the text content used to avoid retrieving relevant scientific and technological documents for the second historical event.

[0055] The accuracy of the judgment results is trained using the loss function Conduct training, including is a historical sequence, including historical events and corresponding descriptions up to the i-th retrieval. The positive sample is the correct answer to the i-th historical event and related sub-questions, including whether to continue the retrieval and the corresponding description of the historical event. The corresponding incorrect answer to the i-th historical event and related sub-questions is a negative sample. The relevant calculation formula is as follows: ;

[0056] in, is a positive sample, is a negative sample, is an adjustable scale parameter, To update the strategy, For reference strategy.

[0057] The RAG retrieval model can be trained using a contrastive learning algorithm. A type of deep learning, the contrastive learning algorithm constructs positive and negative sample pairs to enable the model to learn the discriminative features of the data.

[0058] The contrastive learning algorithm includes the following loss function calculation formula:

[0059] ;

[0060] in, is the preset ratio, S is the text similarity calculation function, q is the query vector, d is the document vector, m is the number of positive samples, and n is the number of negative samples. Positive samples refer to scientific documents corresponding to document vectors that have high similarity with the query vector.

[0061] Based on the first historical event and the preset time range, as well as Construct a query vector, where h is the query vector based on the first historical event, LM represents the embedding model, x represents the relevant scientific documents, n is the negative sample data, and negative samples refer to scientific documents corresponding to document vectors with low similarity to the query vector. d is the document vector.

[0062] The query vector constructed based on historical events and the query vector constructed within a preset time range can be fused to obtain a fused query vector. The corresponding formula is:

[0063] ;

[0064] 、 is the preset ratio, n is the negative sample data, and the negative sample is the scientific document corresponding to the document vector with low similarity to the query vector; h is the query vector of the historical event, is the embedding vector of the preset time range, and d is the document vector.

[0065] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and replacements can be made without departing from the technical principles of the present application. These improvements and replacements should also be regarded as the scope of protection of the present application.

Claims

1. A method for predicting the time series of scientific and technological events based on RAG, characterized in that: The following steps are involved: Establish an event prediction model and train the event prediction model; Establish a RAG retrieval model and train the RAG retrieval model; Obtaining query questions, where the query questions are questions related to scientific and technological event predictions; The trained event prediction model generates a predicted answer based on the query question; the predicted answer includes a judgment label, and the judgment label is used to determine whether it is necessary to use the RAG retrieval model to retrieve the first historical event to generate a description of the first historical event; If retrieval is not required, the event prediction model directly outputs the predicted answer; If retrieval is required, the RAG retrieval model is used to retrieve the first historical event to obtain relevant scientific and technological documents, and a description of the first historical event is obtained based on the relevant scientific and technological documents. The description of the first historical event is added to the historical evidence, and the event prediction model outputs a prediction answer with the added historical evidence; The training of the event prediction model includes training the prediction answer and training the accuracy of the judgment result, the judgment result including the judgment result of whether it is necessary to search for the first historical event, and judging the description of the first historical event and confirming whether it is necessary to search for historical events that occurred before the first historical event; An imitation learning algorithm is used to train the predicted answers, wherein the expert examples include corresponding training sets, the training sets include scientific and technological documents, historical query questions, and answers based on the scientific and technological documents and historical query questions, and the answers based on the scientific and technological documents and historical query questions include: a judgment result on whether it is necessary to search for a first historical event; the predicted content of the scientific and technological event; a judgment result on the description of the first historical event, and confirmation of whether it is necessary to search for historical events that occurred before the first historical event; Use positive and negative samples to train the accuracy of the judgment results; The RAG retrieval model is trained using a contrastive learning algorithm and includes time feature optimization, that is, time features are introduced when calculating vectors.

2. The RAG-based time series prediction method for scientific and technological events according to claim 1, characterized in that: Establish a knowledge base to store scientific and technological documents; Retrieving the first historical event using the RAG retrieval model includes the following steps: using the trained RAG retrieval model to retrieve scientific and technological documents related to the first historical event within a preset time range from the knowledge base, inputting the first historical event and the retrieved scientific and technological documents related to the first historical event into the event prediction model, and generating a description of the first historical event.

3. The RAG-based time series prediction method for scientific and technological events according to claim 2, characterized in that: Retrieving the first historical event further includes: judging the description of the first historical event to determine whether it is necessary to retrieve historical events that occurred before the first historical event; If retrieval is required, a description of a historical event that occurred before the first historical event is generated, and the description of the historical event that occurred before the first historical event is added to the historical evidence, and the event prediction model outputs a prediction answer with the added historical evidence; If retrieval is not required, the retrieval of the first historical event is completed.

4. The RAG-based time series prediction method for scientific and technological events according to claim 2, characterized in that: Retrieving scientific and technological documents related to a first historical event within a preset time range from a knowledge base using a trained RAG retrieval model includes: constructing a query vector based on the first historical event and the preset time range, and adding time features in the process of constructing the query vector; constructing multiple document vectors based on the scientific and technological documents in the knowledge base, and adding time features in the process of constructing the document vector; calculating the similarity between the query vector and the multiple document vectors; obtaining a document vector with high similarity to the query vector based on the similarity between the query vector and the multiple document vectors, and obtaining the scientific and technological documents corresponding to the document vector with high similarity to the query vector.

5. The RAG-based time series prediction method for scientific and technological events according to claim 4 is characterized in that: Embedding models are used to construct query vectors and multiple document vectors.

6. The RAG-based time series prediction method for scientific and technological events according to claim 4, characterized in that: When there are multiple scientific documents corresponding to document vectors with high similarity to the query vector, based on the relevance of the scientific documents to the first historical event and the timeliness of the scientific documents to the preset time range, scientific documents with high relevance and timeliness are selected and input into the event prediction model to generate a description of the first historical event.

Images