Scientific and technological document management method and device, storage medium and computer program product

Abstract

The invention discloses a science and technology document management method and device, a storage medium and a computer program product. Comprising the steps of dividing a science and technology document into a plurality of sentences; the sentences are analyzed through a prediction model, the important scores of the sentences are obtained, when the important scores are larger than a threshold value, the sentences are determined to be key sentences, the important scores are used for quantifying the contribution degree when the sentences reflect the subject of the science and technology document, and the prediction model is obtained by conducting fine adjustment on a first large language model through a first corpus; the key sentences are analyzed through a classification model, the fields to which the key sentences belong are obtained, and the classification model is obtained by conducting fine adjustment on a second large language model through a second corpus; and determining a target field to which the science and technology document belongs according to the field to which each key sentence belongs and the corresponding important score. The technical problem that the classification result is poor in accuracy due to the fact that the context of the document cannot be accurately understood when the science and technology document is classified by a general large model is solved.

Description

Technical Field

[0001] The present application relates to the field of data information processing technology, and more specifically, to a scientific and technological document management method and device, a storage medium, and a computer program product. Background Art

[0002] As the state, enterprises, and scientific research institutions attach more importance to scientific research, various scientific and technological achievements emerge in an endless stream. Accordingly, the number of documents recording scientific and technological achievements (hereinafter referred to as "scientific and technological documents") also increases. Generally, scientific and technological documents are in the form of: patents, academic papers, meeting minutes, computer software copyrights, technical specifications, technical licenses, etc.

[0003] When storing a large number of complex scientific and technological documents, in order to classify and store these scientific and technological documents according to different fields, relevant technicians have proposed classification schemes based on TF-IDF (Term Frequency-Inverse Document Frequency) and naive Bayes classifiers, classification methods based on LDA (Linear Discriminant Analysis) topic models and convolutional neural networks, etc. However, these methods do not fully understand the contextual information of scientific and technological documents, resulting in poor performance in classification accuracy and efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a scientific and technological document management method and device, a storage medium, and a computer program product to at least solve the technical problem that the accuracy of the classification results is poor due to the inability of the general large model to accurately understand the document context when classifying scientific and technological documents.

[0006] According to one aspect of an embodiment of the present application, a method for managing scientific and technological documents is provided, comprising: dividing a scientific and technological document into a plurality of sentences; analyzing each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determining a sentence as a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific and technological document, and the prediction model is obtained by fine-tuning a first large language model using a preset first corpus, and the first corpus includes a plurality of scientific and technological document sentences with annotated importance scores; analyzing a target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second large language model using a preset second corpus, and the second corpus includes scientific and technological document sentences with annotated fields; determining the target field to which the scientific and technological document belongs based on the field to which each key sentence in the scientific and technological document belongs and the corresponding importance score.

[0007] Optionally, the scientific and technological document to be managed is divided into multiple sentences, wherein the scientific and technological document to be managed is obtained and preprocessed, wherein the preprocessing includes at least one of the following: format conversion, data cleaning; the processed scientific and technological document is divided according to preset sentence end symbols to obtain multiple sentences, wherein the sentence end symbols include at least: period, question mark, exclamation mark.

[0008] Optionally, the prediction model includes: an input layer, an encoding layer, and an output layer, wherein the input layer is used to pre-encode the input text to obtain a corresponding first embedding vector sequence, wherein the first embedding vector sequence includes: word embedding vectors of each word in the input text, a position embedding vector used to represent the position information of each word in the input text, and a paragraph embedding vector used to represent the position information of the input text in the original document; the encoding layer includes at least: multiple encoders, multiple dimensional-level attention modules connected to the multiple encoders, each encoder includes a multi-head attention module and a feedforward neural network, wherein the multi-head attention module is used to perform attention calculation on each embedding vector in the first embedding vector sequence to obtain a second embedding vector sequence; the feedforward neural network is used to perform nonlinear changes on the first embedding vector sequence and the second embedding vector sequence to obtain a third embedding vector sequence; each dimensional-level attention module is used to perform weighted processing on the third embedding vector sequence and the embedding vector sequence output by the previous dimensional-level attention module to obtain a fourth embedding vector sequence; the output layer is used to analyze the fourth embedding vector sequence according to a preset weight matrix and bias vector to obtain an important score of the input text.

[0009] Optionally, a pre-trained prediction model is used to analyze the sentence to obtain an importance score of the sentence, including: traversing each sentence in the scientific document, pre-encoding the sentence using the input layer in the prediction model to obtain a corresponding initial embedding vector sequence; analyzing the initial embedding vector sequence using the encoding layer to obtain a corresponding target embedding vector sequence, and analyzing the target embedding vector through the output layer to obtain the importance score of the sentence.

[0010] Optionally, the training process of the prediction model includes: obtaining a first model parameter set and a second model parameter set of a first large language model, wherein the number of model parameters included in the first model parameter set is greater than the number of model parameters included in the second model parameter set; during the training process of the large language model, adjusting each model parameter in the second model parameter set using the first corpus; determining the model parameters of the prediction model based on the first model parameter set and the adjusted second model parameter set; the training process of the classification model includes: obtaining a third model parameter set and a fourth model parameter set of the second large language model, wherein the number of model parameters included in the third model parameter set is greater than the number of model parameters included in the fourth model parameter set; during the training process of the large language model, adjusting each model parameter in the fourth model parameter set using the second corpus; and determining the model parameters of the classification model based on the third model parameter set and the adjusted fourth model parameter set.

[0011] Optionally, the target field to which the scientific document belongs is determined based on the fields to which each key sentence in the scientific document belongs and the corresponding importance scores, including: determining the number of key sentences in each field, and determining the weight value of the corresponding field based on the number of key sentences in the field; taking the weighted sum of the importance scores and corresponding weight values ​​of each key sentence belonging to the same field to obtain the total score of the field; and taking the field with the highest total score as the target field to which the scientific document belongs.

[0012] Optionally, after determining the target field to which the scientific and technological document belongs, the method further comprises: integrating the scientific and technological document into a preset scientific and technological document database according to the target field, wherein the scientific and technological document database includes other scientific and technological documents under multiple fields.

[0013] According to another aspect of an embodiment of the present application, a scientific and technological document management device is also provided, including: a division module, used to divide a scientific and technological document into multiple sentences; a scoring module, used to analyze each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determine that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence when reflecting the theme of the scientific and technological document, and the prediction model is obtained by fine-tuning a first large language model using a preset first corpus, and the first corpus includes multiple scientific and technological document sentences with annotated importance scores; a classification module, used to analyze a target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second large language model using a preset second corpus, and the second corpus includes scientific and technological document sentences with annotated fields; a determination module, used to determine the target field to which the scientific and technological document belongs based on the field to which each key sentence in the scientific and technological document belongs and the corresponding importance score.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned scientific and technological document management method by running the computer program.

[0015] According to another aspect of an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein the computer program implements the above-mentioned scientific and technological document management method when executed by a processor.

[0016] In the embodiment of the present application, by automatically segmenting the scientific and technological documents into sentences, and using the fine-tuned large language model (i.e., prediction model) to encode and analyze each sentence, the key sentences that have important contributions to the document theme are automatically identified, which reduces the workload of manual reading and screening documents and improves the efficiency of information processing. Then, the fine-tuned large language model (i.e., classification model) is used to classify the selected key sentences in the field, ensuring that the classification results of the scientific and technological documents accurately reflect the specific field to which they belong. By comprehensively analyzing the important scores and fields to which each key sentence belongs, the target field to which the scientific and technological document belongs can be accurately identified, so that the documents are automatically classified and integrated into the preset scientific and technological document database, stored by field, which is conducive to the systematic management of knowledge and cross-domain knowledge innovation, and promotes the efficient use and dissemination of scientific and technological information. Among them, since the embodiment of the present application is based on a pre-trained large language model, it has strong adaptability and expansibility, and with the updating and enrichment of the first and second corpora, the accuracy of key sentence screening and field classification is continuously improved, which is suitable for the ever-evolving scientific and technological field. Thereby solving the technical problem that the accuracy of the classification results is poor due to the inability to accurately understand the document context when the general large model classifies the scientific and technological documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of a hardware structure of an optional computer terminal (or mobile device) according to an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an optional scientific and technological document management method according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the architecture of an optional dimension-level attention module according to an embodiment of the present application;

[0021] Figure 4 is a schematic structural diagram of an optional scientific and technological document management device according to an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0026] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0027] Scientific and technological documents: also known as scientific and technological files, are a general term for different forms of documents and materials such as text, graphics, audio and video that record and reflect scientific research, production operations, project construction activities, and equipment and instrument operation, maintenance, and management work.

[0028] Multi-Head Attention Mechanism: Based on the principle of self-attention, it adds multiple attention heads to allocate attention to different dimensions of input information in parallel, thereby capturing richer features and contextual information. Therefore, the specific steps of the multi-head attention mechanism include: first, perform h (h is the number of heads) linear transformations (i.e., weighted and biased) on the vector at each position in the input sequence to divide the input vector into h different subspaces, generate the query matrix Q, key matrix K, and value matrix V; for each subspace, apply the sub-attention calculation attention mechanism, and weight the sum of the value matrix V accordingly to obtain the output of each head:

[0029] head i =Attention(QW i Q ,KW i W ,KW i V )

[0030] Where W i Q ,W i W ,W i V They represent the transformation matrices of the query, key, and value of the i-th head respectively. Finally, the outputs of all heads are concatenated, and then subjected to a final linear transformation and layer normalization to obtain the output of the multi-head attention:

[0031] MultiHeadAttention(Q,K,V)=Concat(head1,…,head h )W O

[0032] Where W O represents the final output transformation matrix, and Concat represents the concatenation operation. Therefore, the multi-head attention mechanism can integrate the information learned from different subspaces and enhance the performance of the model.

[0033] Feed-Forward Neural Network (FFNN): In the encoder or decoder of Transformer, the self-attention layer is followed by a feed-forward neural network, whose main function is to receive the output of the self-attention layer and perform further nonlinear transformations on it to capture more complex features and representations. Among them, the feed-forward neural network includes two linear transformations and uses a nonlinear activation function (such as RuLU) between them. The first layer of linear transformation is a fully connected layer, which is used to map the output of the self-attention layer to a higher-dimensional space and introduce nonlinearity; and the second layer of linear transformation is also a fully connected layer, which is used to map the output of the previous layer back to the same dimension (or compatible dimension) as the input, usually without introducing additional nonlinearity. Therefore, the output of the feed-forward neural network can be expressed as:

[0034]

[0035] In the formula It is the result of adding the output of the i-th self-attention layer in the k-th encoder to the input. W1, W2 represent the weight matrix of the feedforward neural network, and b1, b2 represent the bias term of the feedforward neural network.

[0036] TF-IDF (TermFrequency-Inverse Document Frequency): is a method used to evaluate the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but decreases in inverse proportion to the frequency of its appearance in the corpus. Among them, TF (TermFrequency) represents the frequency of occurrence of a term in a text, and IDF (InverseDocument Frequency) can be obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of the quotient.

[0037] Example 1

[0038] According to an embodiment of the present application, an embodiment of a method for managing scientific and technological documents is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a scientific and technological document management method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0041] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the scientific and technological document management method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the scientific and technological document management method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] In the above operating environment, Figure 2 is a flow chart of an optional scientific and technological document management method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S208, wherein:

[0045] Step S202, dividing the scientific document into a plurality of sentences.

[0046] In the technical solution provided in step S202, the above-mentioned scientific and technological documents can be academic papers, meeting minutes, computer software copyright, integrated circuit layout design, detection methods, technical specifications, technical know-how, technology use licenses, etc. Because sentences are the basic units of natural language, they carry complete grammatical structures and semantic information. Compared to processing entire paragraphs or entire documents, the analysis of sentence level allows the model to focus on smaller information units, thereby improving the accuracy of key information identification. Therefore, the present application embodiment will first divide scientific and technological documents into multiple complete sentences, so as to convert complex large-scale text analysis problems into a series of smaller, more manageable sub-problems. This usually means lower computational complexity and faster processing speed for machine learning models.

[0047] Step S204, using the pre-trained prediction model to analyze each sentence, obtain the importance score of each sentence, and determine the sentence as a key sentence when the importance score is greater than a preset threshold value.

[0048] In the technical solution provided in step S204, each sentence in the scientific document is traversed, and for each sentence, the sentence is encoded using a prediction model to obtain a corresponding target embedding vector sequence, wherein the target embedding vector sequence not only contains global context information, but also highlights the characteristics of specific aspects in the sentence. The prediction model analyzes the target embedding vector sequence of the sentence to predict the importance score of the sentence, wherein the importance score can quantify the contribution of the sentence to the theme of the scientific document. According to the relationship between the importance score of the sentence and the preset threshold value, the sentence is determined to be a key sentence, wherein: when the importance score is greater than the preset threshold value, the sentence is determined to be a key sentence; conversely, when the importance score is not greater than the preset threshold value, the sentence is determined not to be a key sentence. In addition, in order to better meet the specific needs of evaluating the importance of sentences in scientific and technological documents, the embodiment of the present application uses a preset first corpus to fine-tune the first language model, wherein the first language model can be any model in the pre-trained model library, such as the BERT (Bidirectional Encoder Representation from Transformers) model, and the first corpus includes a plurality of scientific and technological document sentences with annotated importance scores.

[0049] Step S206: Analyze the target embedding vector sequence of each key sentence using the pre-trained classification model to obtain the field to which each key sentence belongs.

[0050] In the technical solution provided in step S206, after identifying the key sentences in the scientific document, the classification model is then used to analyze the target embedding vector sequence of each key sentence to obtain the field to which each key sentence belongs, wherein the field includes but is not limited to: energy field (such as thermal power, hydropower, wind power, solar energy, etc.), communication field (such as wireless communication, optical fiber communication, etc.), etc. In addition, in order to better meet the specific needs of analyzing scientific documents, the embodiment of the present application uses a preset second corpus to fine-tune the second largest language model, wherein the second largest language model can be any model in the pre-trained model library, or it can be the same large model as the first largest language model, such as the BERT model; the second corpus includes sentences of scientific documents with annotated fields.

[0051] Step S208, determining the target field to which the scientific document belongs according to the fields to which each key sentence in the scientific document belongs and the corresponding importance scores.

[0052] In the technical solution provided in step S208, by accumulating the importance scores of all key sentences in each field and summing up the weighted importance scores of all fields, a total score is calculated for each field, which represents the relevance strength and importance level of the document in each field. Then, the total scores of all fields are compared, and the field with the highest score is selected as the target field of the scientific and technological document. This decision-making process comprehensively considers the field distribution of key sentences in the document and their importance, ensuring that the determination of the target field is based on an in-depth analysis of the overall content of the document, rather than just a simple match based on keywords or sentences.

[0053] The above method of this embodiment is further introduced below.

[0054] As an optional implementation, in the technical solution provided in the above step S202, the scientific and technological documents may be divided according to the following steps, including:

[0055] Step S2021, obtaining the scientific and technological documents to be managed, and preprocessing the scientific and technological documents, wherein the preprocessing includes at least one of the following: format conversion, data cleaning.

[0056] Specifically, scientific and technological document data can be obtained through public channels such as the Internet and private channels of enterprises. Since scientific and technological documents may be in PDF, image or other non-plain text formats, it is first necessary to convert the obtained scientific and technological documents into plain text format through format conversion tools (such as optical character recognition), and delete special characters, page numbers and other irrelevant symbols, extra spaces, tabs and other irrelevant format information, footnotes, chart text descriptions and other irrelevant text content in the scientific and technological documents in plain text format, so as to obtain the cleaned scientific and technological document text data.

[0057] Step S2022, dividing the processed scientific and technological document according to preset sentence ending symbols to obtain multiple sentences.

[0058] In the technical solution provided in the above step S2022, regular expressions or sentence segmenters of built-in NLP libraries (such as NLTK, spaCy) can be used to perform preliminary sentence segmentation based on common sentence ending symbols such as periods, question marks, exclamation marks, etc.

[0059] It should be noted that in order to avoid the situation where the sentence segmentation errors occur due to the direct use of sentence end symbols to segment scientific documents, the embodiment of the present application can further use a pre-trained deep learning model (such as a long short-term memory network LSTM, a gated recurrent unit GRU or BERT, etc.) to optimize the initial sentence segmentation results (i.e., the multiple text fragments obtained) obtained in the above step S2022. The specific process is as follows: the preliminary segmented text fragments are input into the model, and the model will generate a feature representation of each word based on the context information and word embedding; then, the model predicts whether there is a sentence boundary after each word. Among them, the deep learning model has been trained with a large amount of text data, and it can recognize more complex text structures, such as professional expressions or sentence structures in scientific documents; finally, according to the output of the model, the sentence boundaries are readjusted to obtain multiple complete sentences in the scientific document.

[0060] As an optional implementation, in the technical solution provided in the above step S204, the training process of the prediction model includes: obtaining a first model parameter set and a second model parameter set of a first large language model, wherein the number of model parameters included in the first model parameter set is greater than the number of model parameters included in the second model parameter set; during the training process of the large language model, adjusting each model parameter in the second model parameter set using the first corpus; and determining the model parameters of the prediction model based on the first model parameter set and the adjusted second model parameter set.

[0061] That is to say, during the fine-tuning process, the first large language model keeps most of its original model parameters unchanged to protect the extensive language comprehension and generation capabilities of the original large language model; at the same time, the first corpus is used to continue to iteratively train the first large model to adjust a small part of the model parameters; finally, the first model parameter set and the adjusted second model parameter set are used as model parameters of the prediction model to obtain the above-mentioned prediction model, so that the prediction model can not only maintain the original language capabilities, but also analyze the importance of sentences in scientific and technological documents, thereby showing higher accuracy and professionalism in evaluating the importance of sentences in scientific and technological documents.

[0062] In addition, the above prediction model includes: an input layer, a coding layer, and an output layer, wherein:

[0063] The input layer is used to process the input text S i Pre-coding is performed to obtain the corresponding first embedded vector sequence X i , where the first embedding vector sequence X i Included: word embedding vector C of each word in the input text i , used to represent the position information of each word in the input text position embedding vector P iand the paragraph embedding vector E used to represent the position information of the input text in the original document i .

[0064] Specifically, the word embedding (Token Embedding) module in the input layer can divide the preprocessed input text into a series of words or words (tokens), and each word or subword is mapped to a multidimensional word embedding vector. In addition, since the Transformer architecture does not have an inherent sequential processing mechanism such as a recurrent neural network (RNN), and different lexical positions may produce different semantic information, the position information of words is crucial for the model to understand word order and context. Therefore, in the embodiment of the present application, the position information of the word in the sentence is also encoded in the form of a sine wave or a cosine wave, and the positional embedding (Positional Embedding) vector is used to ensure that the model can learn the relative position information of the word, not just the absolute position. In order to distinguish whether the input text comes from different paragraphs or text fragments, the embodiment of the present application also adds the segment embedding (Segment Embedding) vector to the word embedding vector to reflect the segmentation information of the word in the text, wherein, in models such as BERT, paragraph embedding is also referred to as "segment ID" or "sentence type embedding". For example, when processing a text concatenated from two sentences A and B, all words belonging to sentence A will be assigned one paragraph embedding vector, while words belonging to sentence B will be assigned another different paragraph embedding vector.

[0065] In the specific implementation, the input text is converted into a sequence of word IDs, and then the word embedding vector is obtained by looking up the word embedding matrix; the position embedding vector is directly looked up from the position embedding matrix; the paragraph embedding vector can be obtained based on the segmentation information of the input text. Finally, these three vectors are added at the element level to form the final embedding vector. This pre-coding process ensures the integrity of the input information and the richness of the context, which is the key to the model's efficient understanding of text.

[0066] The encoding layer includes at least: multiple stacked encoders, multiple stacked dimension-level attention modules (i.e., a deep attention network) connected to the multiple encoders, wherein the dimension level specifically refers to the various feature dimensions that constitute the embedding vector sequence, so that the model can identify and emphasize the most relevant parts of these dimensional features to the task (such as sentence importance evaluation, classification, etc.). Specifically, each encoder includes two sublayers: a multi-head attention module and a feedforward neural network. Among them, the multi-head attention module is used to perform attention calculations on each embedding vector in the first embedding vector sequence to obtain a second embedding vector sequence; the feedforward neural network is used to perform nonlinear changes on the first embedding vector sequence and the second embedding vector sequence to obtain a third embedding vector sequence to capture more complex features and representations; each dimension-level attention module is used to perform weighted processing on the third embedding vector sequence and the embedding vector sequence output by the previous dimension-level attention module to obtain a fourth embedding vector sequence. Among them, the above-mentioned dimension-level attention module is a network architecture constructed based on the DSE algorithm (DimAttention-basedSentence Embedding Algorithm), and its specific structure is as follows Figure 3 As shown, therefore, the calculation expression of the dimension-level attention module in the embodiment of the present application can be:

[0067]

[0068] Among them, the j-th layer DimAttentionLayer j The input is the j-1th layer DimAttentionLayer j-1 Output.

[0069] The output layer (i.e., the linear layer) is used to analyze the fourth embedding vector sequence according to the preset weight matrix and bias vector to determine the input text S according to the number of seed words in the original document contained in the fourth embedding vector sequence. iThe seed word is the top m segmented words with the highest number of appearances of all segmented words in the original document according to TF-IDF statistics. The determination method includes: calculating the quotient of the number of appearances of each segmented word in the original document and the total number of segmented words in the original document in advance, and taking the quotient as the word frequency of the segmented word, that is, TF = the number of appearances of a certain segmented word in the original document / the total number of segmented words in the original document; then, calculating the total number of documents in the first corpus divided by the number of documents containing the segmented word, and taking the logarithm of the quotient to obtain the inverse frequency of the segmented word. Document frequency, i.e., IDF = log (total number of documents in the first corpus / number of documents of the sub-word + 1); then, calculate the term frequency of a single sub-word - inverse document frequency TF-IDF = term frequency TF * inverse document frequency IDF; finally, take the first m sub-words with the highest TF-IDF as the seed words of the original document. The reason is that for the TF-IDF of each sub-word calculated, the larger the TF-IDF, the higher the degree of discrimination of the sub-word for the original document, and the sub-word with a larger TF-IDF value is taken as the seed word of the original document.

[0070] Therefore, the output layer can calculate the input text S according to the following formula i The importance score of the input text S i Its relative importance within the document and its contribution to the subject and content of the document):

[0071] Y i =σ(W l ·E i +b l )

[0072] Where W l represents the weight matrix of the linear layer, b l Represents the bias term, σ represents the activation function, such as the sigmoid function, and its expression is as follows:

[0073]

[0074] Therefore, according to the prediction model of the above structure, it can analyze the sentences in the scientific document in the following process to obtain the corresponding importance score: traverse each sentence in the scientific document, use the input layer in the prediction model to pre-encode the sentence, and obtain the corresponding initial embedding vector sequence; use the encoding layer to analyze the initial embedding vector sequence to obtain the corresponding target embedding vector sequence, and analyze the target embedding vector through the output layer to obtain the importance score of the sentence. Among them, the higher the importance score, the more important the sentence plays a core and key role in the scientific document.

[0075] Furthermore, a threshold value of the importance of the key sentence score is predefined. If the importance score of the sentence is greater than the threshold value, the sentence is determined to be a key sentence in the scientific document. For each key sentence, the target embedding vector sequence of the key sentence is analyzed using a pre-trained classification model to identify the technical field to which the key sentence belongs.

[0076] As an optional implementation, in the technical solution provided in the above step S206, the training process of the classification model includes: obtaining a third model parameter set and a fourth model parameter set of the second largest language model, wherein the number of model parameters included in the third model parameter set is greater than the number of model parameters included in the fourth model parameter set; during the training process of the large language model, adjusting each model parameter in the fourth model parameter set using the second corpus; and determining the model parameters of the classification model based on the third model parameter set and the adjusted fourth model parameter set.

[0077] It should be noted that the training principle and structure of the classification model are similar to those of the above-mentioned prediction model, the only difference being that the results of the model output are different. This application will not further explain the training principle and structure of the classification model.

[0078] Therefore, the embodiment of the present application can use the above-mentioned pre-trained classification model to analyze the target embedding vector sequence of key sentences to obtain the corresponding fields, including: traversing each key sentence in the scientific document, using the input layer in the classification model to pre-encode the sentence to obtain the corresponding initial embedding vector sequence; using the encoding layer to analyze the initial embedding vector sequence to obtain the corresponding target embedding vector sequence, and analyzing the target embedding vector through the output layer to obtain the probability distribution of the sentence belonging to each field, and taking the field with the highest probability as the classification result of the key sentence.

[0079] It should be noted that in steps S204-S206, the large language model (including the first large language model and the second large language model) can be loaded into the memory. For example, the raw data of the large language model can be loaded from the non-volatile memory into the volatile memory so that the processor can run the large language model. The raw data of the large language model refers to unprocessed data, which usually includes parameters and structural data of the large language model. The structural data can be a calculation relationship based on the parameters, such as the forward propagation calculation relationship between the intermediate layers and the neurons. Specifically, the structural data can include the structure-related code of the large language model, such as the code for performing related calculations between the intermediate layers and the neurons.

[0080] In one embodiment, an area for loading a large language model can be divided in the memory, which may include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of a specific parameter in the parameter storage area through a pointer. During the training process of the large language model, it may be necessary to frequently update the parameters, and the parameter values ​​in the parameter storage area can be updated.

[0081] As an optional implementation, in the technical solution provided in the above step S208, the target field to which the scientific and technological document belongs may be determined according to the following method, including:

[0082] Step S2081, determine the number of key sentences in each field, and determine the weight value of the corresponding field according to the number of key sentences in the field. It should be noted that the above weight value can also be pre-set according to the importance of the field.

[0083] Step S2082, weighted sum of the importance scores and corresponding weight values ​​of each key sentence in the same field is performed to obtain the total score of the field. This weighting process ensures that the evaluation of the overall content of the document is not only based on the number of sentences, but also takes into account the importance and field relevance of the sentences, and the obtained weighted result (i.e., the total score) represents the relevance strength and importance level of the scientific and technological document in various fields.

[0084] Step S2083, the domain with the highest total score is used as the target domain to which the scientific document belongs. This decision-making process comprehensively considers the domain distribution of key sentences in the document and their importance, ensuring that the determination of the target domain is based on an in-depth analysis of the overall content of the document, rather than simply based on the simple matching of keywords or sentences.

[0085] As another optional implementation, in the technical solution provided in the above step S208, the target field to which the scientific document belongs can also be determined according to the following method, including: determining the classification score of the field to which the scientific document belongs based on the importance scores of each key sentence in the same field; and taking the field with the highest classification score as the target field to which the scientific document belongs.

[0086] Furthermore, after determining the target field to which the scientific and technological document belongs, in order to better manage the scientific and technological document, the embodiment of the present application also proposes that the scientific and technological document can be integrated into a preset scientific and technological document database according to the target field.

[0087] The scientific and technological document database includes sub-databases corresponding to multiple fields, and each sub-database stores multiple documents in the corresponding field. By storing scientific and technological documents in different fields in an organized manner, when users or systems need to find documents in a specific field, they can directly access the sub-database in that field, avoiding searching in a large and unclassified document collection, thereby saving time and computing resources.

[0088] Based on the scheme defined by the above-mentioned steps S202 to S208, it can be learned that in an embodiment, by automatically segmenting the scientific and technological documents into sentences, and using a fine-tuned large language model (i.e., a prediction model) to encode and analyze each sentence, the key sentences that contribute significantly to the document theme are automatically identified, reducing the workload of manual reading and screening documents, and improving the efficiency of information processing. Then, the fine-tuned large language model (i.e., a classification model) is used to classify the selected key sentences in the field, ensuring that the classification results of the scientific and technological documents accurately reflect the specific field to which they belong. By comprehensively analyzing the important scores and the fields to which each key sentence belongs, the target field to which the scientific and technological document belongs can be accurately identified, thereby automatically classifying the documents and integrating them into a preset scientific and technological document database, storing them by field, which is conducive to the systematic management of knowledge and cross-domain knowledge innovation, and promoting the efficient use and dissemination of scientific and technological information. Among them, since the embodiment of the present application is based on a pre-trained large language model, it has strong adaptability and expansibility, and with the updating and enrichment of the first and second corpora, the accuracy of key sentence screening and field classification is continuously improved, and it is suitable for the ever-evolving scientific and technological field.

[0089] It can be seen that through the technical solution of the embodiment of the present application, advanced natural language processing technology can be effectively utilized to realize intelligent management and in-depth analysis of scientific and technological documents, bringing significant improvements and conveniences to scientific research, knowledge management, decision support and other fields.

[0090] Example 2

[0091] Based on the first embodiment of the present application, an embodiment of a scientific and technological document management device is also provided, and the scientific and technological document management method of the above embodiment is executed when the device is running. Figure 4 is a schematic diagram of the structure of an optional scientific and technological document management device according to an embodiment of the present application, such as Figure 4 As shown, the scientific and technological document management device at least includes a division module 42, a scoring module 44, a classification module 46 and a determination module 48, wherein:

[0092] A division module 42, for dividing the scientific document into a plurality of sentences;

[0093] A scoring module 44 is used to analyze each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determine that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning the first language model using a preset first corpus, and the first corpus includes a plurality of scientific document sentences with annotated importance scores;

[0094] A classification module 46 is used to analyze the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning the second language model using a preset second corpus, and the second corpus includes sentences of scientific and technological documents with annotated fields;

[0095] The determination module 48 is used to determine the target field to which the scientific document belongs based on the fields to which each key sentence in the scientific document belongs and the corresponding importance scores.

[0096] It should be noted that the various modules in the above-mentioned scientific and technological document management device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0097] Example 3

[0098] According to an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the scientific and technological document management method in Example 1.

[0099] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: dividing the scientific document into multiple sentences; analyzing each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determining the sentence as a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning a first language model using a preset first corpus, and the first corpus includes multiple scientific document sentences with annotated importance scores; analyzing the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second language model using a preset second corpus, and the second corpus includes scientific document sentences with annotated fields; determining the target field to which the scientific document belongs based on the field to which each key sentence in the scientific document belongs and the corresponding importance score.

[0100] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein the computer program implements the scientific and technological document management method in Example 1 when executed by a processor.

[0101] Optionally, the computer program executes the following steps: dividing the scientific document into multiple sentences; analyzing each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determining that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence to the theme of the scientific document, and the prediction model is obtained by fine-tuning a first language model using a preset first corpus, and the first corpus includes multiple scientific document sentences with annotated importance scores; analyzing the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second language model using a preset second corpus, and the second corpus includes scientific document sentences with annotated fields; determining the target field to which the scientific document belongs based on the field to which each key sentence in the scientific document belongs and the corresponding importance score.

[0102] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the scientific and technological document management method in Example 1 is executed when the program is running.

[0103] Optionally, the following steps are executed when the program is running: dividing the scientific document into multiple sentences; analyzing each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determining the sentence as a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning a first language model using a preset first corpus, and the first corpus includes multiple scientific document sentences with annotated importance scores; analyzing the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second language model using a preset second corpus, and the second corpus includes scientific document sentences with annotated fields; determining the target field to which the scientific document belongs based on the field to which each key sentence in the scientific document belongs and the corresponding importance score.

[0104] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 5 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 5As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the scientific and technological document management method in the above-mentioned embodiment 1 when running.

[0105] Optionally, the processor is configured to execute the following steps through a computer program: dividing the scientific document into multiple sentences; analyzing each sentence using a pre-trained prediction model to obtain an importance score for each sentence, and determining that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning a first language model using a preset first corpus, and the first corpus includes multiple scientific document sentences with annotated importance scores; analyzing the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning a second language model using a preset second corpus, and the second corpus includes scientific document sentences with annotated fields; determining the target field to which the scientific document belongs based on the field to which each key sentence in the scientific document belongs and the corresponding importance score.

[0106] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0112] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A scientific and technological document management method, characterized in that: include: Divide scientific documents into sentences; Analyze each of the sentences using a pre-trained prediction model to obtain an importance score for each of the sentences, and determine that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning a first language model using a preset first corpus, and the first corpus includes a plurality of scientific document sentences with annotated importance scores; Analyzing each of the key sentences using a pre-trained classification model to obtain the field to which each of the key sentences belongs, wherein the classification model is obtained by fine-tuning the second largest language model using a preset second corpus, and the second corpus includes sentences of scientific and technological documents with annotated fields; The target field to which the scientific document belongs is determined according to the fields to which each key sentence in the scientific document belongs and the corresponding importance scores.

2. The method according to claim 1, characterized in that Divide the scientific and technological documents to be managed into multiple sentences, among which, Acquire the scientific and technological documents to be managed, and pre-process the scientific and technological documents, wherein the pre-processing includes at least one of the following: format conversion, data cleaning; The processed scientific and technological document is divided according to preset sentence ending symbols to obtain the plurality of sentences, wherein the sentence ending symbols at least include: a period, a question mark, and an exclamation mark.

3. The method according to claim 1, characterized in that The prediction model includes: an input layer, a coding layer, and an output layer, wherein: The input layer is used to pre-encode the input text to obtain a corresponding first embedding vector sequence, wherein the first embedding vector sequence includes: word embedding vectors of each word in the input text, position embedding vectors used to represent position information of each word in the input text, and paragraph embedding vectors used to represent position information of the input text in the original document; The encoding layer at least includes: multiple encoders, multiple dimension-level attention modules connected to the multiple encoders, each of the encoders includes a multi-head attention module and a feedforward neural network, wherein the multi-head attention module is used to perform attention calculation on each embedding vector in the first embedding vector sequence to obtain a second embedding vector sequence; the feedforward neural network is used to perform nonlinear changes on the first embedding vector sequence and the second embedding vector sequence to obtain a third embedding vector sequence; each of the dimension-level attention modules is used to perform weighted processing on the third embedding vector sequence and the embedding vector sequence output by the previous dimension-level attention module to obtain a fourth embedding vector sequence; The output layer is used to analyze the fourth embedding vector sequence according to a preset weight matrix and bias vector to obtain an importance score of the input text.

4. The method according to claim 3, characterized in that: The sentence is analyzed using a pre-trained prediction model to obtain an importance score for the sentence, including: Each of the sentences in the scientific document is traversed, and the sentences are pre-encoded using the input layer in the prediction model to obtain a corresponding initial embedding vector sequence; the initial embedding vector sequence is analyzed using the encoding layer to obtain a corresponding target embedding vector sequence, and the target embedding vector is analyzed through the output layer to obtain an importance score of the sentence.

5. The method according to claim 1, characterized in that The training process of the prediction model includes: obtaining a first model parameter set and a second model parameter set of the first large language model, wherein the number of model parameters included in the first model parameter set is greater than the number of model parameters included in the second model parameter set; during the training process of the large language model, adjusting each model parameter in the second model parameter set using the first corpus; and determining the model parameters of the prediction model according to the first model parameter set and the adjusted second model parameter set; The training process of the classification model includes: obtaining a third model parameter set and a fourth model parameter set of the second large language model, wherein the number of model parameters included in the third model parameter set is greater than the number of model parameters included in the fourth model parameter set; during the training process of the large language model, adjusting each model parameter in the fourth model parameter set using the second corpus; and determining the model parameters of the classification model based on the third model parameter set and the adjusted fourth model parameter set.

6. The method according to claim 1, characterized in that Determining the target field to which the scientific document belongs based on the fields to which each key sentence in the scientific document belongs and the corresponding importance scores includes: Determine the number of key sentences in each of the fields, and determine the weight value of the corresponding field according to the number of key sentences in the field; Performing a weighted summation of the importance scores and corresponding weight values ​​of each key sentence in the same field to obtain a total score of the field; The field with the highest total score is taken as the target field to which the scientific and technological document belongs.

7. The method according to claim 1, characterized in that After determining the target field to which the scientific and technological document belongs, the method further comprises: The scientific and technological documents are integrated into a preset scientific and technological document database according to the target field, wherein the scientific and technological document database includes other scientific and technological documents under multiple fields.

8. A scientific and technological document management device, characterized in that: include: A segmentation module, used to segment scientific documents into multiple sentences; A scoring module, used for analyzing each of the sentences using a pre-trained prediction model to obtain an importance score for each of the sentences, and determining that the sentence is a key sentence when the importance score is greater than a preset threshold value, wherein the importance score is used to quantify the contribution of the sentence in reflecting the theme of the scientific document, and the prediction model is obtained by fine-tuning the first language model using a preset first corpus, and the first corpus includes a plurality of scientific document sentences with annotated importance scores; A classification module, used to analyze the target embedding vector sequence of each key sentence using a pre-trained classification model to obtain the field to which each key sentence belongs, wherein the classification model is obtained by fine-tuning the second largest language model using a preset second corpus, and the second corpus includes sentences of scientific and technological documents with annotated fields; The determination module is used to determine the target field to which the scientific document belongs based on the fields to which each key sentence in the scientific document belongs and the corresponding importance scores.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the scientific and technological document management method described in any one of claims 1 to 7 by running the computer program.

10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the method for managing scientific and technological documents as claimed in any one of claims 1 to 7 is implemented.

Images