Technical trend prediction method and system based on multi-layer fine tuning of large language model

Abstract

The invention relates to the technical field of large language models, in particular to a technology trend prediction method and system based on multi-layer fine tuning of a large language model. Performing clustering processing on the data after the time sequence preprocessing based on a clustering algorithm to divide the result data into n categories, and taking three technical keywords with highest frequency in each category as technical tags to obtain clustering data; performing knowledge graph processing on the data subjected to the time sequence preprocessing to obtain an association relationship among the technical points, and obtaining text description content according to the association relationship among the technical points to obtain knowledge graph data; and according to the data after the time sequence preprocessing, the clustering data and the knowledge graph data, carrying out fine tuning on the pre-trained large language model to obtain a technology prediction large model, and carrying out technology trend prediction through the technology prediction large model. The prediction accuracy of the large language model can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a technology trend prediction method and system based on multi-layer fine-tuning of a large language model. Background Art

[0002] Technology forecasting is the pre-speculation or determination of the future goals, possible paths and resource conditions of scientific and technological development. Technology forecasting can not only provide a basis for technical functional departments to formulate scientific and technological policies and development strategies, select priority development areas, and determine R&D funding, but also provide technical information for industrial technology upgrades and enterprise development strategies, guide R&D personnel to carry out research and development activities, widely attract manpower and funds from all walks of life, increase the R&D efforts of potential technology projects, promote cooperation and exchanges among all aspects of society, and strengthen the communication capabilities between the government, research institutions, enterprises and universities, thus forming a vibrant technology innovation network.

[0003] Traditional technology forecasting is mainly achieved through expert interviews and market research, but with the explosion of knowledge and data, the method of relying on manpower to carry out technology forecasting has encountered many challenges, especially in interdisciplinary fields. In recent years, technology forecasting based on data mining has become a new research hotspot. By collecting and analyzing a large amount of data, using statistics and machine learning algorithms, the laws and trends hidden behind the data are excavated. However, due to the quality of data and the complexity of knowledge reasoning, the effect of technology forecasting is still not satisfactory.

[0004] Therefore, how to utilize information such as external knowledge bases and databases to further improve the performance of technology prediction requires a more advanced system and method to solve.

[0005] In recent years, with the rapid development and popularization of large language models, their rich background knowledge and massive data training models, especially the rapid development of large model fine-tuning technology, have provided new possibilities for technology prediction based on massive technical data. Fine-tuning of large models is defined as the process of further training the model using new, task-specific data sets based on a pre-trained large deep learning model. The main purpose of this fine-tuning technology is to enable the model to adapt to new, specific tasks or fields without having to train a completely new model from scratch.

[0006] The advantage of large model fine-tuning is that it can make full use of the common features of the pre-trained model and quickly adapt to new task requirements based on a small amount of new data. This technology not only improves the training efficiency of the model, but also reduces the dependence on large-scale labeled data.

[0007] Pre-trained models are usually very versatile and can perform well on multiple tasks. However, due to differences in data distribution and characteristics between different tasks, it is often difficult to achieve optimal performance by directly using pre-trained models. Therefore, it is necessary to further train the model on task-specific datasets through fine-tuning techniques to adapt to new task requirements.

[0008] In practical applications, models are often required to be customized and optimized for specific scenarios or tasks. Large model fine-tuning is a technology that was born to meet this demand. Through fine-tuning, the model can be made more adaptable to new tasks or fields while maintaining the powerful feature extraction capabilities of the pre-trained model, thereby improving the practicality and performance of the model. Summary of the invention

[0009] The present invention overcomes the shortcoming of low prediction accuracy of existing large language models and provides a technology trend prediction method and system based on multi-layer fine-tuning of a large language model.

[0010] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is:

[0011] The first aspect of the present invention discloses a technology trend prediction method based on multi-layer fine-tuning of a large language model, comprising the following steps:

[0012] Acquire scientific research achievement data based on a big data network, construct a fine-tuned original data set according to the scientific research achievement data, and perform time series preprocessing on the fine-tuned original data set to obtain time series preprocessed data;

[0013] Based on the clustering algorithm, the data after time series preprocessing is clustered to divide the achievement data into n categories, and the three most frequently appearing technical keywords in each category are used as technical tags to obtain clustered data;

[0014] Perform knowledge graph processing on the data after time series preprocessing to obtain the correlation between technical points, obtain text description content based on the correlation between technical points, and obtain knowledge graph data;

[0015] The pre-trained large language model is fine-tuned according to the time series preprocessed data, clustering data and knowledge graph data to obtain a technology prediction large model, and technology trend prediction is performed through the technology prediction large model.

[0016] Among them, the big data of scientific research results are divided based on time periods, and the period T is defined as one year or three years.

[0017] More specifically, clustering is performed on the time series preprocessed data based on a clustering algorithm to divide the achievement data into n categories, and the three most frequently appearing technical keywords in each category are used as technical tags to obtain clustered data, specifically:

[0018] Each result data in the time series preprocessed data is regarded as a separate object, and each object contains various feature information related to the technology; each object is vectorized according to the feature information of each object to obtain the feature vector of each object;

[0019] Treat each object as a separate class, calculate the similarity between each two classes based on the feature vectors of each object, and merge the two classes with the highest similarity into a new class. Repeat this merging process, and recalculate the similarity between the new class and other classes each time you merge.

[0020] As the merging proceeds, the number of classes decreases gradually, and a hierarchical clustering result is gradually formed. In this process, the clustering status is continuously monitored until the predetermined number of classes n is reached;

[0021] After obtaining n categories, extract technical keywords for each category, traverse all the results data in each category, and count the frequency of technical keywords appearing in them;

[0022] Finally, the three most frequently occurring technical keywords were selected from each category as the technical labels of the corresponding category to obtain clustered data.

[0023] More specifically, the data after time series preprocessing is processed into a knowledge graph to obtain the association relationship between technical points, and the text description content is obtained according to the association relationship between technical points to obtain the knowledge graph data, which is specifically:

[0024] Obtain the technical text of each scientific and technological achievement based on the time series preprocessed data, extract keywords from the technical text of each scientific and technological achievement based on natural language processing technology, identify words with a frequency greater than a preset frequency threshold, and use the words with a frequency greater than the preset frequency threshold as technical keywords;

[0025] The extracted technical keywords are used to construct a knowledge graph. Each technical keyword is regarded as a node in the knowledge graph. Then, the semantic relationship between technical keywords in the technical text is analyzed, including causal relationship, parallel relationship and belonging relationship. According to the semantic relationship between technical keywords, edges are established between corresponding nodes, thereby constructing a preliminary knowledge graph structure.

[0026] Introducing a graph neural network training algorithm, and optimizing the representation of nodes and edges in the knowledge graph structure according to the graph neural network training algorithm, so as to obtain the correlation relationship between technical points;

[0027] Extracting graph-related node information according to the acquired association relationship between the technical points, wherein the graph-related node information includes the identifier of the node itself and the relationship type between the nodes;

[0028] The graph-associated node information is input as prompt content into the pre-trained large language model. The pre-trained large language model processes the input associated node information based on its pre-trained knowledge and language comprehension ability, and converts the association relationship between technical points into text description content in natural language to form knowledge graph data.

[0029] More specifically, the pre-trained large language model is fine-tuned based on the time series preprocessed data, clustering data, and knowledge graph data to obtain a technical prediction large model, specifically:

[0030] Inputting the time series preprocessed data, clustering data and knowledge graph data into a pre-trained large language model;

[0031] A comprehensive fine-tuning method is used to fine-tune the time series preprocessed data in the pre-trained large language model; a supervised fine-tuning method is used to fine-tune the clustering data set in the pre-trained large language model; a parameter fine-tuning method is used to fine-tune the knowledge graph data in the pre-trained large language model;

[0032] After completing the fine-tuning step, the technical data of all cycles are integrated according to the preset requirements; the pre-trained large language model is predicted based on the integrated data to obtain the prediction results; the prediction results include the new technology names or keywords that may appear in the next cycle;

[0033] Calculate the prediction error based on the prediction results and the actual situation; update the model weights based on the prediction error through the back propagation algorithm; among them, back propagation propagates the prediction error from the output layer of the model to the input layer, and adjusts the various parameters of the model based on the prediction error, so that the model's ability to predict the next cycle technology is continuously improved;

[0034] Repeat the prediction and weight update process until the model reaches the preset prediction accuracy in predicting the technology of the next cycle, and then build a large prediction model based on technology forecasting.

[0035] More specifically, a comprehensive fine-tuning method is used to fine-tune the time series preprocessed data in the pre-trained large language model, specifically:

[0036] The time series preprocessed data is sequentially input into the pre-trained large language model. Since the time series data contains information about the development of technology in the time dimension, the model comprehensively learns the characteristics of this data, including the evolution trend of technology over time and the changes in technology in different time periods.

[0037] Genetic algorithms are introduced and all parameters of the model are fully adjusted based on genetic algorithms to adapt to the characteristics of time series data;

[0038] After each iteration, monitor the learning effect of the model on the time series data, and predict the accuracy of the technical trend of the next time period by calculating the loss function to determine whether the model has been effectively trained on the time series data;

[0039] If the learning effect of the model on the time series data does not meet the preset requirements, the parameters of the model will continue to be adjusted based on the genetic algorithm until the learning effect of the model on the time series data meets the preset requirements.

[0040] More specifically, a supervised fine-tuning method is used to fine-tune the clustering dataset in the pre-trained large language model, specifically:

[0041] Randomly select several cluster data as samples. Since the cluster data has been divided into n categories and has corresponding technical labels, the samples are marked using the information of each technical label. The marked content reflects the category feature information of the cluster.

[0042] The labeled clustering dataset is input into the pre-trained large language model. The model learns based on the input samples and their labeled information and adjusts the parameters within the model. In this process, the feature learning related to the clustering category is focused on, so that the model can understand the technical features and differences represented by different clustering categories.

[0043] The loss function is calculated based on the difference between the model's output and the labeled true value. The loss function reflects the prediction error of the model on the clustering dataset.

[0044] Using the machine gradient descent optimization algorithm, the model parameters are fine-tuned according to the gradient direction of the loss function, so that the prediction ability of the model on the clustering data set is gradually improved until the prediction ability meets the preset requirements.

[0045] More specifically, the parameter fine-tuning method is used to fine-tune the knowledge graph data in the pre-trained large language model, specifically:

[0046] Preprocess the knowledge graph data to make it conform to the format requirements of the model input, including converting the graph-related node information and the relationship between technical points into a vector representation that the model can understand;

[0047] The preprocessed knowledge graph data is input into the pre-trained large language model, and the model uses its parameter fine-tuning mechanism to adjust some parameters related to the knowledge graph data instead of retraining the entire model.

[0048] Obtain the accuracy of the relationship between the predicted technical points, and determine the performance indicators of the model on the knowledge graph data based on the accuracy of the relationship between the predicted technical points;

[0049] If the performance indicators do not meet expectations, the structural parameters of the knowledge graph data are adjusted according to the feedback of the performance indicators, and the model's processing capabilities for the knowledge graph data are optimized until the performance indicators meet expectations.

[0050] The second aspect of the present invention discloses a technology trend prediction system based on multi-layer fine-tuning of a large language model, which is applied to any one of the technology trend prediction methods based on multi-layer fine-tuning of a large language model, and the technology trend prediction system comprises:

[0051] The time series preprocessing module is responsible for dividing the scientific research results big data based on time periods;

[0052] The clustering labeling module is responsible for labeling the technical types or technical points of scientific research results through clustering algorithms to form technical points or technical descriptions;

[0053] The knowledge graph construction module is responsible for extracting technical keywords from the technical text of each scientific and technological achievement and then constructing a knowledge graph;

[0054] The knowledge graph training and generation module is responsible for training the knowledge graph to obtain the correlation between technical points, and then inputs the graph-related node information as prompt content into the large model. Through the large language model, the correlation between technical points is generated into text description content.

[0055] The technology trend prediction system also includes:

[0056] The comprehensive fine-tuning module is responsible for comprehensively fine-tuning the time series preprocessed data and updating all parameters of the pre-trained model to make full use of the common features of the pre-trained model.

[0057] The supervised fine-tuning module is responsible for fine-tuning the parameters of the pre-trained large model according to the training instructions of the scheduling module based on the clustering data, and training the model using labeled data and unlabeled data;

[0058] The parameter fine-tuning module is responsible for fine-tuning the parameters of the pre-trained large model based on the knowledge graph dataset and the training instructions of the scheduling module;

[0059] The fine-tuning scheduling module is responsible for scheduling the training of the fine-tuning dataset.

[0060] The present invention solves the technical defects existing in the background technology, and the present invention has the following beneficial effects: scientific research results data is obtained based on a big data network, a fine-tuned original data set is constructed according to the scientific research results data, and the fine-tuned original data set is subjected to time series preprocessing to obtain the time series preprocessed data; the time series preprocessed data is clustered based on a clustering algorithm to divide the results data into n categories, and the three most frequently appearing technical keywords in each category are used as technical labels to obtain clustered data; the time series preprocessed data is subjected to knowledge graph processing to obtain the association relationship between technical points, and the text description content is obtained according to the association relationship between technical points to obtain knowledge graph data; the pre-trained large language model is fine-tuned according to the time series preprocessed data, the clustered data and the knowledge graph data to obtain a large model for technology prediction, and the technology trend prediction is performed through the large model for technology prediction. The present invention can effectively improve the prediction accuracy of the large language model, and provide strong support for the prediction of technical research directions and technical development planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying creative work.

[0062] Figure 1 This is a flow chart of the first method of the technical trend prediction method;

[0063] Figure 2 This is a flow chart of the second method of the technical trend prediction method;

[0064] Figure 3 This is the system block diagram of this technology trend prediction system. DETAILED DESCRIPTION

[0065] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0067] like Figure 1As shown, the first aspect of the present invention discloses a technology trend prediction method based on multi-layer fine-tuning of a large language model, comprising the following steps:

[0068] S102: acquiring scientific research achievement data based on a big data network, constructing a fine-tuning original data set according to the scientific research achievement data, and performing time series preprocessing on the fine-tuning original data set to obtain time series preprocessed data;

[0069] It should be noted that the big data of scientific research results are divided into time periods, and the period T is defined as one or three years. Taking into account the changes in the speed of technological development, a dynamic period can be adopted. For example, a period of 10 years is used before 1990, a period of 5 years is used from 1990 to 2010, and a period of 2 years is used from 2010 to date.

[0070] S104: clustering the data after time series preprocessing based on a clustering algorithm to divide the achievement data into n categories, and taking the three most frequently appearing technical keywords in each category as technical tags to obtain clustered data;

[0071] It should be noted that clustering is performed on the data after time series preprocessing, and the technology type or technology point is labeled through the clustering algorithm. For example, the technical text of each scientific and technological achievement is represented, such as vectorized by Roberta, and then the technical achievements are divided into n categories through the k-means algorithm, hierarchical clustering algorithm, etc., and the three most frequently appearing technical keywords in each category are used as technology labels.

[0072] S106: Perform knowledge graph processing on the data after time series preprocessing to obtain the association relationship between technical points, obtain text description content according to the association relationship between technical points, and obtain knowledge graph data;

[0073] It should be noted that the data after time series preprocessing is processed by knowledge graph. Specifically, technical keywords are extracted from the technical text of each scientific and technological achievement, and then a knowledge graph is constructed and trained to obtain the correlation between technical points. The graph related node information is then input into the big model as prompt content, and the correlation between technical points is generated into text description content through the big language model.

[0074] S108: Fine-tune the pre-trained large language model according to the time series preprocessed data, clustering data and knowledge graph data to obtain a technology prediction large model, and perform technology trend prediction through the technology prediction large model.

[0075] It should be noted that through fine-tuning of the above three types of data sets, the technical data of all previous cycles are used to predict the new technology names or keywords of the next cycle, and the model weights are updated through back propagation to build a large model based on technology prediction. So far, a model construction method and system based on fine-tuning of a large language model is designed to predict future technology research directions.

[0076] More specifically, clustering is performed on the time series preprocessed data based on a clustering algorithm to divide the achievement data into n categories, and the three most frequently appearing technical keywords in each category are used as technical tags to obtain clustered data, specifically:

[0077] Each result data in the time series preprocessed data is regarded as a separate object, and each object contains various feature information related to the technology; each object is vectorized according to the feature information of each object to obtain the feature vector of each object;

[0078] Treat each object as a separate class, calculate the similarity between each two classes based on the feature vectors of each object, and merge the two classes with the highest similarity into a new class. Repeat this merging process, and recalculate the similarity between the new class and other classes each time you merge.

[0079] As the merging proceeds, the number of classes decreases gradually, and a hierarchical clustering result is gradually formed. In this process, the clustering status is continuously monitored until the predetermined number of classes n is reached;

[0080] After obtaining n categories, extract technical keywords for each category, traverse all the results data in each category, and count the frequency of technical keywords appearing in them;

[0081] Finally, the three most frequently occurring technical keywords were selected from each category as the technical labels of the corresponding category to obtain clustered data.

[0082] It should be noted that, first, each result data in the time series preprocessed data is treated as a separate object, and these objects contain various technical-related feature information. This step is the basis for subsequent operations, clarifies the data unit to be processed, and emphasizes the importance of the technical characteristics contained in each result data itself. According to the feature information of each object, each object is vectorized to obtain the feature vector of each object. Vectorization is to convert the complex feature information of the object into a numerical form that can be easily processed by the computer, which can be better used for operations such as calculating the similarity between objects. Initially, each object is regarded as a separate class, and then the similarity between each two classes is calculated based on their feature vectors. Then, the two classes with the highest similarity are merged into a new class, and the similarity between the new class and other classes is recalculated after each merger. This process is repeated continuously, and the number of classes will gradually decrease, forming a hierarchical clustering result. In this process, the clustering status is continuously monitored until the predetermined number of categories n is reached. This hierarchical clustering method can gradually build a clustering structure according to the similarity of data objects, taking into account the internal relationship between data, and adapting to the overall clustering goal by continuously adjusting the relationship between classes. When the clustering results reach n categories, technical keywords are extracted for each category. All the results data in each category are traversed to count the frequency of technical keywords appearing in them. Finally, the three technical keywords with the highest frequency of occurrence are selected from each category as the technical labels of the corresponding category. This step can dig out the most representative technical features from each clustering category and summarize the core technical content of each category in a concise way.

[0083] Through the above series of operations based on clustering algorithms, the achievement data after time series preprocessing can be effectively clustered to form n categories. This clustering method fully considers the similarity of technical features in the achievement data and mines the inherent structure of the data through vectorization and hierarchical clustering processes. The technical labels for each category further summarize the core technical content of each category. The obtained clustered data not only reflects the classification of the achievement data, but also concisely identifies the technical focus of each category through the technical labels.

[0084] More specifically, the data after time series preprocessing is processed by knowledge graph to obtain the association relationship between technical points, and the text description content is obtained according to the association relationship between technical points to obtain knowledge graph data, such as Figure 2 As shown, specifically:

[0085] S202: Obtain the technical text of each scientific and technological achievement according to the time series preprocessed data, extract keywords from the technical text of each scientific and technological achievement based on natural language processing technology, identify words with a frequency greater than a preset frequency threshold, and use the words with a frequency greater than the preset frequency threshold as technical keywords;

[0086] S204: constructing a knowledge graph using the extracted technical keywords, treating each technical keyword as a node in the knowledge graph, and then analyzing the semantic relationship between the technical keywords in the technical text, including causal relationship, parallel relationship, and belonging relationship, and establishing edges between corresponding nodes according to the semantic relationship between the technical keywords, thereby constructing a preliminary knowledge graph structure;

[0087] S206: Introducing a graph neural network training algorithm, and optimizing the representation of nodes and edges in the knowledge graph structure according to the graph neural network training algorithm, so as to obtain the correlation relationship between technical points;

[0088] S208: extracting graph-related node information according to the acquired association relationship between the technical points, wherein the graph-related node information includes the identifier of the node itself and the relationship type between the nodes;

[0089] S210: Input the graph-associated node information as prompt content into the pre-trained large language model. The pre-trained large language model processes the input associated node information based on its pre-trained knowledge and language comprehension ability, and converts the association relationship between technical points into text description content in natural language to form knowledge graph data.

[0090] It should be noted that the technical text of each scientific and technological achievement is first obtained from the time series preprocessed data. Then, keyword extraction is performed based on natural language processing technology. A preset frequency threshold is set here, and words with a frequency greater than the threshold are used as technical keywords. This step extracts relatively important words from the technical text by frequency screening. These words can represent the key concepts in the technical achievements to a certain extent. The knowledge graph is constructed using the extracted technical keywords. Each technical keyword is regarded as a node in the knowledge graph, and then the semantic relationship between the technical keywords in the technical text is analyzed, such as causal relationship, parallel relationship and belonging relationship. According to these semantic relationships, edges are established between the corresponding nodes to construct a preliminary knowledge graph structure. This construction method can present the concepts and their relationships in the technical achievements in a structured way, which is helpful to explore the deep connection between technical points. The graph neural network training algorithm is introduced to optimize the representation of nodes and edges in the knowledge graph structure. Through this training, the correlation between technical points can be obtained more accurately. The graph neural network can learn and optimize the structural information in the knowledge graph, so that the representation of nodes and edges can better reflect the real relationship between technical points. Extract the graph-associated node information based on the acquired association relationship between technical points. This information includes the node's own identification and the type of relationship between nodes. This step is to extract key information from the knowledge graph so that it can be subsequently input into the pre-trained large language model. Input the graph-associated node information into the pre-trained large language model as prompt content. The pre-trained large language model processes the input associated node information with its pre-trained knowledge and language understanding ability, converts the association relationship between technical points into a text description content in natural language, and finally forms the knowledge graph data. This step converts the structured relationship in the knowledge graph into a natural language form that can be understood by humans, facilitating further analysis and application.

[0091] Through this series of operations, we can build a knowledge graph from the time series preprocessed data, obtain the associations between technical points, and finally convert them into natural language text descriptions to form knowledge graph data. This method makes full use of natural language processing technology and graph neural network training algorithms to mine key concepts and their relationships in technical achievements and present them in the form of natural language.

[0092] More specifically, the pre-trained large language model is fine-tuned based on the time series preprocessed data, clustering data, and knowledge graph data to obtain a technical prediction large model, specifically:

[0093] Inputting the time series preprocessed data, clustering data and knowledge graph data into a pre-trained large language model;

[0094] A comprehensive fine-tuning method is used to fine-tune the time series preprocessed data in the pre-trained large language model; a supervised fine-tuning method is used to fine-tune the clustering data set in the pre-trained large language model; a parameter fine-tuning method is used to fine-tune the knowledge graph data in the pre-trained large language model;

[0095] After completing the fine-tuning step, the technical data of all cycles are integrated according to the preset requirements; the pre-trained large language model is predicted based on the integrated data to obtain the prediction results; the prediction results include the new technology names or keywords that may appear in the next cycle;

[0096] Calculate the prediction error based on the prediction results and the actual situation; update the model weights based on the prediction error through the back propagation algorithm; among them, back propagation propagates the prediction error from the output layer of the model to the input layer, and adjusts the various parameters of the model based on the prediction error, so that the model's ability to predict the next cycle technology is continuously improved;

[0097] Repeat the prediction and weight update process until the model reaches the preset prediction accuracy in predicting the technology of the next cycle, and then build a large prediction model based on technology forecasting.

[0098] It should be noted that first, the time series preprocessed data, clustering data, and knowledge graph data are input into the pre-trained large language model. Then different fine-tuning methods are used for different types of data. After completing the fine-tuning step, the technical data of all cycles are integrated according to the preset requirements. The pre-trained large language model makes predictions based on the integrated data to obtain prediction results containing the names of new technologies or keywords that may appear in the next cycle. This process integrates data from different sources so that the model can comprehensively consider multiple technical information for prediction. The prediction error is calculated based on the prediction results and the actual situation (if there is some known technical information for the next cycle). Through the back propagation algorithm, the prediction error is propagated from the output layer of the model to the input layer, and the various parameters of the model are adjusted according to the error. This back propagation mechanism can optimize the model according to the prediction deviation, so that the model's ability to predict the next cycle technology is continuously improved. Repeat the prediction and weight update process until the model reaches the preset prediction accuracy in predicting the next cycle technology, and then the prediction large model based on technology prediction is built. This process continuously optimizes the model to ensure that the model's prediction ability meets the requirements. By integrating technical data from all periods for prediction, combining the calculation of prediction errors and the back-propagation algorithm to update model weights, the model's prediction capabilities are continuously optimized, thereby effectively improving the model's prediction accuracy.

[0099] More specifically, a comprehensive fine-tuning method is used to fine-tune the time series preprocessed data in the pre-trained large language model, specifically:

[0100] The time series preprocessed data is sequentially input into the pre-trained large language model. Since the time series data contains information about the development of technology in the time dimension, the model comprehensively learns the characteristics of this data, including the evolution trend of technology over time and the changes in technology in different time periods.

[0101] Genetic algorithms are introduced and all parameters of the model are fully adjusted based on genetic algorithms to adapt to the characteristics of time series data;

[0102] After each iteration, monitor the learning effect of the model on the time series data, and predict the accuracy of the technical trend of the next time period by calculating the loss function to determine whether the model has been effectively trained on the time series data;

[0103] If the learning effect of the model on the time series data does not meet the preset requirements, the parameters of the model will continue to be adjusted based on the genetic algorithm until the learning effect of the model on the time series data meets the preset requirements.

[0104] It should be noted that the pre-processed time series data is first input into the pre-trained large language model in sequence. Because time series data contains information about the development of technology in the time dimension, the model can fully learn the characteristics of these data. This includes the evolution trend of technology over time, such as the process of technology from germination to maturity, and the changes in technology in different time periods, such as the innovation of technology, the expansion of application scope or the improvement of performance in a specific time period. This comprehensive learning helps the model to deeply understand the dynamic process of technology development. The introduction of genetic algorithms to fully adjust all parameters of the model aims to make the model better adapt to the characteristics of time series data. Genetic algorithms are an optimization algorithm that simulates natural selection and genetic mechanisms. It can search for the optimal solution in the parameter space. By adjusting all parameters of the model through genetic algorithms, the complex relationship between time series data and model parameters can be excavated, so that the model can better fit the laws of time series data. After each iteration cycle is completed, the learning effect of the model on the time series data is monitored. Here, the accuracy of the technology trend in the next time period is predicted by calculating the loss function to determine whether the model is effectively trained on the time series data. The loss function is an indicator that measures the difference between the model's predicted value and the true value. It evaluates the model's learning effect by predicting the accuracy of the technology trend in the next time period, which can directly reflect the model's mastery of the technology development laws in the time series data. If the model's learning effect on the time series data does not meet the preset requirements, continue to adjust the model's parameters based on the genetic algorithm until the model's learning effect on the time series data meets the preset requirements. This process ensures that the model can fully learn the technology development laws in the time series data and continuously improve the model's fitting and prediction capabilities for time series data. It enables the model to deeply explore the technology development laws in the time series data, including the evolution trend of technology over time and technology changes in different time periods, thereby improving the model's accuracy in predicting the technology trend in the next time period, and allowing the model to perform better when processing time-related technology prediction tasks.

[0105] More specifically, a supervised fine-tuning method is used to fine-tune the clustering dataset in the pre-trained large language model, specifically:

[0106] Randomly select several cluster data as samples. Since the cluster data has been divided into n categories and has corresponding technical labels, the samples are marked using the information of each technical label. The marked content reflects the category feature information of the cluster.

[0107] The labeled clustering dataset is input into the pre-trained large language model. The model learns based on the input samples and their labeled information and adjusts the parameters within the model. In this process, the feature learning related to the clustering category is focused on, so that the model can understand the technical features and differences represented by different clustering categories.

[0108] The loss function is calculated based on the difference between the model's output and the labeled true value. The loss function reflects the prediction error of the model on the clustering dataset.

[0109] Using the machine gradient descent optimization algorithm, the model parameters are fine-tuned according to the gradient direction of the loss function, so that the prediction ability of the model on the clustering data set is gradually improved until the prediction ability meets the preset requirements.

[0110] It should be noted that, first, a number of cluster data are randomly selected as samples. Since the cluster data has been divided into n categories and has corresponding technical labels, the samples are labeled using these technical label information. The labeled content reflects the category feature information of the cluster. This step provides a clear goal for the supervised learning of the subsequent model, that is, to let the model learn the mapping relationship between the cluster category features and the labels. The labeled cluster data set is input into the pre-trained large language model. The model learns according to the input samples and their label information, and adjusts the parameters inside the model in this process. In particular, the model will focus on feature learning related to the cluster category, which helps the model to deeply understand the technical features and differences represented by different cluster categories. For example, the model can identify the differences between different categories of technology in terms of function, application field or technical complexity. The loss function is calculated based on the difference between the output of the model and the true value of the label. The loss function is an important indicator to measure the prediction accuracy of the model. In this case, it reflects the prediction error of the model on the cluster data set. If the output of the model is significantly different from the true label value, the value of the loss function will be larger, otherwise it will be smaller. Using the stochastic gradient descent optimization algorithm, fine-tune the model parameters according to the gradient direction of the loss function. The stochastic gradient descent algorithm is a commonly used optimization algorithm that gradually improves the model's predictive ability on clustered data sets by continuously adjusting parameters in the direction where the loss function decreases fastest. This process will continue until the model's predictive ability on clustered data sets meets the preset requirements, ensuring that the model can accurately classify clustered data or predict information related to cluster categories. Paying attention to cluster category-related features during model learning, and fine-tuning the model parameters in combination with the calculation of the loss function and the stochastic gradient descent optimization algorithm can effectively improve the model's predictive ability for clustered data sets, allowing the model to better understand the technical characteristics and differences represented by different cluster categories, thereby providing a more accurate model foundation for subsequent technical analysis, prediction and other tasks based on clustering information.

[0111] More specifically, the parameter fine-tuning method is used to fine-tune the knowledge graph data in the pre-trained large language model, specifically:

[0112] Preprocess the knowledge graph data to make it conform to the format requirements of the model input, including converting the graph-related node information and the relationship between technical points into a vector representation that the model can understand;

[0113] The preprocessed knowledge graph data is input into the pre-trained large language model, and the model uses its parameter fine-tuning mechanism to adjust some parameters related to the knowledge graph data instead of retraining the entire model.

[0114] Obtain the accuracy of the relationship between the predicted technical points, and determine the performance indicators of the model on the knowledge graph data based on the accuracy of the relationship between the predicted technical points;

[0115] If the performance indicators do not meet expectations, the structural parameters of the knowledge graph data are adjusted according to the feedback of the performance indicators, and the model's processing capabilities for the knowledge graph data are optimized until the performance indicators meet expectations.

[0116] It should be noted that the knowledge graph data is first preprocessed. The purpose of this step is to make it conform to the format requirements of the model input. Specifically, the graph association node information, the association relationship between technical points, etc. are converted into a vector representation that the model can understand. This vector representation can be processed by the pre-trained large language model, which is a necessary step to integrate the knowledge graph data into the model, because the model usually processes data based on vector operations. The pre-processed knowledge graph data is input into the pre-trained large language model. The model uses its parameter fine-tuning mechanism to only adjust some parameters related to the knowledge graph data, rather than retraining the entire model. The advantage of this is that it can not only use the information in the knowledge graph data to optimize the model's processing ability for knowledge graph-related tasks, but also retain the knowledge and capabilities of the pre-trained model in other aspects, avoiding the waste of time and resources caused by large-scale retraining. Obtain the accuracy of predicting the association relationship between technical points, and determine the performance indicators of the model on the knowledge graph data based on this accuracy. This performance indicator directly reflects the model's ability to predict the association relationship between technical points in the knowledge graph data, and is the key basis for evaluating the performance of the model on knowledge graph-related tasks. If the performance indicators do not meet expectations, the structural parameters related to the knowledge graph data are adjusted based on the feedback of the performance indicators. Through this adjustment, the model's ability to process knowledge graph data can be optimized, so that the model can better process information such as the relationship between technical points in the knowledge graph. This process will continue until the performance indicators meet expectations, ensuring that the model has good performance on the knowledge graph data. On the basis of retaining the advantages of the pre-trained model, it can effectively utilize the information in the knowledge graph data, improve the model's ability to predict the relationship between technical points in the knowledge graph, enable the model to meet the expected performance requirements when processing knowledge graph-related tasks, and provide more accurate model support for subsequent technical analysis and prediction work involving knowledge graph data.

[0117] The second aspect of the present invention discloses a technology trend prediction system based on multi-layer fine-tuning of a large language model, which is applied to any of the technology trend prediction methods based on multi-layer fine-tuning of a large language model, such as Figure 3 As shown, the technology trend prediction system includes:

[0118] Time series preprocessing module: responsible for dividing the scientific research results big data based on time periods;

[0119] Clustering labeling module: responsible for labeling the technical types or technical points of scientific research results through clustering algorithms to form technical points or technical descriptions;

[0120] Knowledge graph construction module: responsible for extracting technical keywords from the technical text of each scientific and technological achievement, and then constructing a knowledge graph;

[0121] Knowledge graph training and generation module: responsible for training the knowledge graph to obtain the relationship between technical points, and then input the graph related node information as prompt content into the big model, and generate text description content from the relationship between technical points through the big language model;

[0122] Comprehensive fine-tuning module: responsible for comprehensively fine-tuning the time series preprocessed data and updating all parameters of the pre-trained model to make full use of the common features of the pre-trained model;

[0123] Supervised fine-tuning: Responsible for fine-tuning the parameters of the pre-trained large model according to the training instructions of the scheduling module, using labeled data and unlabeled data to train the model, and improving the performance of the model through semi-supervised learning algorithms or adversarial learning algorithms;

[0124] Parameter Efficient Fine-tuning Module: Responsible for fine-tuning the parameters of the pre-trained large model according to the training instructions of the scheduling module based on the knowledge graph dataset. Adapter Tuning: Insert small neural network modules (adapters) between specific layers of the pre-trained model and train only these adapters. Parameter Efficient Fine-tuning (PEFT) technology achieves the goal of reducing the fine-tuning cost while maintaining the performance of the pre-trained model by minimizing the number of fine-tuning parameters and computational complexity.

[0125] Fine-tuning scheduling module: responsible for scheduling the training of fine-tuning data sets, using the technical data of all previous cycles to predict the new technology name or keywords of the next cycle, updating the model weights through back propagation, and building a large model based on technology prediction; Pre-trained large language model system: pre-trained large models that can be fine-tuned. Common pre-trained models include BERT, GPT, RoBERTa, etc., which are suitable for different natural language processing (NLP) tasks.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0127] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0129] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0130] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0131] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A technology trend prediction method based on multi-layer fine-tuning of a large language model, characterized in that: The following steps are involved: Acquire scientific research achievement data based on a big data network, construct a fine-tuned original data set according to the scientific research achievement data, and perform time series preprocessing on the fine-tuned original data set to obtain time series preprocessed data; Based on the clustering algorithm, the data after time series preprocessing is clustered to divide the achievement data into n categories, and the three most frequently appearing technical keywords in each category are used as technical tags to obtain clustered data; Perform knowledge graph processing on the data after time series preprocessing to obtain the correlation between technical points, obtain text description content based on the correlation between technical points, and obtain knowledge graph data; The pre-trained large language model is fine-tuned according to the time series preprocessed data, clustering data and knowledge graph data to obtain a technology prediction large model, and technology trend prediction is performed through the technology prediction large model.

2. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 1, characterized in that: The big data of scientific research results are divided into time periods, and the period T is defined as one or three years.

3. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 1 is characterized in that: Based on the clustering algorithm, the data after time series preprocessing is clustered to divide the achievement data into n categories, and the three most frequently appearing technical keywords in each category are used as technical tags to obtain clustered data, specifically: Each result data in the time series preprocessed data is regarded as a separate object, and each object contains various feature information related to the technology; each object is vectorized according to the feature information of each object to obtain the feature vector of each object; Treat each object as a separate class, calculate the similarity between each two classes based on the feature vectors of each object, and merge the two classes with the highest similarity into a new class. Repeat this merging process, and recalculate the similarity between the new class and other classes each time you merge. As the merging proceeds, the number of classes decreases gradually, and a hierarchical clustering result is gradually formed. In this process, the clustering status is continuously monitored until the predetermined number of classes n is reached; After obtaining n categories, extract technical keywords for each category, traverse all the results data in each category, and count the frequency of technical keywords appearing in them; Finally, the three most frequently occurring technical keywords were selected from each category as the technical labels of the corresponding category to obtain clustered data.

4. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 1 is characterized in that: The data after time series preprocessing is processed by knowledge graph to obtain the association relationship between technical points. The text description content is obtained according to the association relationship between technical points to obtain knowledge graph data, which is specifically: Obtain the technical text of each scientific and technological achievement based on the time series preprocessed data, extract keywords from the technical text of each scientific and technological achievement based on natural language processing technology, identify words with a frequency greater than a preset frequency threshold, and use the words with a frequency greater than the preset frequency threshold as technical keywords; The extracted technical keywords are used to construct a knowledge graph. Each technical keyword is regarded as a node in the knowledge graph. Then, the semantic relationship between technical keywords in the technical text is analyzed, including causal relationship, parallel relationship and belonging relationship. According to the semantic relationship between technical keywords, edges are established between corresponding nodes, thereby constructing a preliminary knowledge graph structure. Introducing a graph neural network training algorithm, and optimizing the representation of nodes and edges in the knowledge graph structure according to the graph neural network training algorithm, so as to obtain the correlation relationship between technical points; Extracting graph-related node information according to the acquired association relationship between the technical points, wherein the graph-related node information includes the identifier of the node itself and the relationship type between the nodes; The graph-associated node information is input as prompt content into the pre-trained large language model. The pre-trained large language model processes the input associated node information based on its pre-trained knowledge and language comprehension ability, and converts the association relationship between technical points into text description content in natural language to form knowledge graph data.

5. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 1 is characterized in that: The pre-trained large language model is fine-tuned based on the time series preprocessed data, clustering data, and knowledge graph data to obtain the technical prediction large model, specifically: Inputting the time series preprocessed data, clustering data and knowledge graph data into a pre-trained large language model; A comprehensive fine-tuning method is used to fine-tune the time-series preprocessed data in the pre-trained large language model; Use supervised fine-tuning methods to fine-tune the clustering dataset in the pre-trained large language model; Use parameter fine-tuning method to fine-tune the knowledge graph data in the pre-trained large language model; After completing the fine-tuning step, integrate the technical data of all cycles according to the preset requirements; The pre-trained large language model makes predictions based on the integrated data to obtain prediction results, which include new technology names or keywords that may appear in the next cycle. Calculate the prediction error based on the prediction results and the actual situation; update the model weights based on the prediction error through the back propagation algorithm; among them, back propagation propagates the prediction error from the output layer of the model to the input layer, and adjusts the various parameters of the model based on the prediction error, so that the model's ability to predict the next cycle technology is continuously improved; Repeat the prediction and weight update process until the model reaches the preset prediction accuracy in predicting the technology of the next cycle, and then build a large prediction model based on technology forecasting.

6. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 5, characterized in that: A comprehensive fine-tuning method is used to fine-tune the time series preprocessed data in the pre-trained large language model. Specifically: The time series preprocessed data is sequentially input into the pre-trained large language model. Since the time series data contains information about the development of technology in the time dimension, the model comprehensively learns the characteristics of this data, including the evolution trend of technology over time and the changes in technology in different time periods. Genetic algorithms are introduced and all parameters of the model are fully adjusted based on genetic algorithms to adapt to the characteristics of time series data; After each iteration, monitor the learning effect of the model on the time series data, and predict the accuracy of the technical trend of the next time period by calculating the loss function to determine whether the model has been effectively trained on the time series data; If the learning effect of the model on the time series data does not meet the preset requirements, the parameters of the model will continue to be adjusted based on the genetic algorithm until the learning effect of the model on the time series data meets the preset requirements.

7. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 5, characterized in that: The supervised fine-tuning method is used to fine-tune the clustering dataset in the pre-trained large language model. Specifically: Randomly select several cluster data as samples. Since the cluster data has been divided into n categories and has corresponding technical labels, the samples are marked using the information of each technical label. The marked content reflects the category feature information of the cluster. The labeled clustering dataset is input into the pre-trained large language model. The model learns based on the input samples and their labeled information and adjusts the parameters within the model. In this process, attention is paid to feature learning related to clustering categories, so that the model can understand the technical features and differences represented by different clustering categories. The loss function is calculated based on the difference between the model's output and the labeled true value. The loss function reflects the prediction error of the model on the clustering dataset. Using the machine gradient descent optimization algorithm, the model parameters are fine-tuned according to the gradient direction of the loss function, so that the prediction ability of the model on the clustering data set is gradually improved until the prediction ability meets the preset requirements.

8. The technology trend prediction method based on multi-layer fine-tuning of a large language model according to claim 5 is characterized in that: The parameter fine-tuning method is used to fine-tune the knowledge graph data in the pre-trained large language model. Specifically: Preprocess the knowledge graph data to make it conform to the format requirements of the model input, including converting the graph-related node information and the relationship between technical points into a vector representation that the model can understand; The preprocessed knowledge graph data is input into the pre-trained large language model, and the model uses its parameter fine-tuning mechanism to adjust some parameters related to the knowledge graph data instead of retraining the entire model. Obtain the accuracy of the relationship between the predicted technical points, and determine the performance indicators of the model on the knowledge graph data based on the accuracy of the relationship between the predicted technical points; If the performance indicators do not meet expectations, the structural parameters of the knowledge graph data are adjusted according to the feedback of the performance indicators, and the model's processing capabilities for the knowledge graph data are optimized until the performance indicators meet expectations.

9. A technology trend prediction system based on multi-layer fine-tuning of a large language model, applied to a technology trend prediction method based on multi-layer fine-tuning of a large language model as claimed in any one of claims 1 to 8, characterized in that: The technology trend prediction system comprises: The time series preprocessing module is responsible for dividing the scientific research results big data based on time periods; The clustering labeling module is responsible for labeling the technical types or technical points of scientific research results through clustering algorithms to form technical points or technical descriptions; The knowledge graph construction module is responsible for extracting technical keywords from the technical text of each scientific and technological achievement and then constructing a knowledge graph; The knowledge graph training and generation module is responsible for training the knowledge graph to obtain the correlation between technical points, and then inputs the graph-related node information as prompt content into the large model. Through the large language model, the correlation between technical points is generated into text description content.

10. A technology trend prediction system based on multi-layer fine-tuning of a large language model according to claim 9, characterized in that: The technology trend prediction system also includes: The comprehensive fine-tuning module is responsible for comprehensively fine-tuning the time series preprocessed data and updating all parameters of the pre-trained model to make full use of the common features of the pre-trained model. The supervised fine-tuning module is responsible for fine-tuning the parameters of the pre-trained large model according to the training instructions of the scheduling module based on the clustering data, and training the model using labeled data and unlabeled data; The parameter fine-tuning module is responsible for fine-tuning the parameters of the pre-trained large model based on the knowledge graph dataset and the training instructions of the scheduling module; The fine-tuning scheduling module is responsible for scheduling the training of the fine-tuning dataset.

Images