Biomedical text pre-training generation method

Abstract

The invention discloses a biomedical text pre-training generation method, which comprises the following steps of: dividing an input biomedical text sentence into smaller fragments or word blocks, and performing processing through mark embedding, fragment embedding and position embedding to obtain a pre-processed text; generating a text embedding vector from the preprocessed text through a self-attention mechanism; the text embedded vectors are clustered by adopting a clustering algorithm, finally all the embedded vectors are divided into clusters, and each cluster is represented by a centroid; selecting m vectors closest to the center in each cluster, and arranging the m vectors according to an appearance sequence in the document; and the selected context embedding vector is coded by BertSum, decoding is carried out through six layers of Random Transformers, and abstract extraction is converted into abstract generation. According to the method, the higher semantic similarity between the extracted text and the source text is ensured, and the overall semantic consistency between the candidate abstracts and the original document is emphasized.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and biomedicine, and in particular to a biomedical text pre-training generation method. Background Art

[0002] The importance of biomedical text generation lies in its ability to significantly improve the efficiency of information retrieval and knowledge management in the biomedical and research fields. As the volume of biomedical literature continues to grow rapidly, more and more researchers and healthcare professionals need to deal with large amounts of complex information. Automatically generating and summarizing biomedical text can access key information more quickly and accurately, supporting key tasks such as clinical decision-making, scientific reviews, and drug development. In addition, biomedical text generation can help answer complex biomedical questions, generate new research hypotheses, and promote the dissemination and application of biomedical knowledge. In summary, biomedical text generation plays an indispensable role in improving information accessibility, accelerating research progress, and supporting precision medicine, and is crucial to the advancement of the entire biomedical field. Summary of the invention

[0003] The purpose of the present invention is to provide a biomedical text pre-training generation method.

[0004] The technical solution adopted by the present invention is:

[0005] A biomedical text pre-training generation method, comprising the following steps:

[0006] S1, uses the WordPiece tokenization algorithm to divide the input biomedical text sentences into smaller fragments or chunks, which are then processed through token embedding, fragment embedding, and position embedding;

[0007] S2, the preprocessed text is processed by the Self-Attention mechanism in the Transformer model to generate a text embedding vector;

[0008] S3, using K-means clustering algorithm to embed text vectors Clustering is performed, and the optimization goal is to minimize the Euclidean distance between the embedding vector and the centroid of the cluster to which it belongs, and the centroid is updated iteratively until convergence, and finally k clusters are obtained, each cluster is represented by a centroid;

[0009] S4, select the vectors closest to the center in each cluster, each vector represents a sentence in the text; arrange these sentences in the order they appear in the document;

[0010] S5, encode the selected context embedding vectors by BertSum and decode them through six layers of RandomTransformer, convert the extracted summary into a generated summary, thus completing the biomedical text generation process.

[0011] Further, the specific method of step S1 is as follows:

[0012] S101, divide the input biomedical text sentences into smaller segments or chunks, add a special [CLS] token at the beginning of each biomedical sentence to represent the information of the whole sentence, use a [SEP] token at the end to represent the sentence boundary, and finally represent the sentence T as a token sequence (t1, t2, … t n );

[0013] S102, process each sequence (t1, t2, … t n ) in the sentence T, convert each sequence into a vector e(t i ) = TokenEmbedding(t i ) through token embedding,

[0014] then convert it into a vector s(t i ) = SegmentEmbedding(t i ) through segment embedding,

[0015] finally assign a position embedding p(t i ) = PositionEmbedding(t i ) to each sequence;

[0016] The representation of the final sequence is:

[0017] S103, perform the processing of S101 and S102 on all sentences in the text to obtain the preprocessed vector H 0 .

[0018] Further, the specific method of step S2 is as follows:

[0019] S201, perform a linear transformation on the input sequence vectors through three weight matrices to obtain: query vector Q = H 0 * W Q (W Q is the weight matrix of Q), key vector K = H 0 * W K (W K is the weight matrix of K), value vector V = H 0 * W V (W Vis the weight matrix of V); then multiply the query vector (Q) of each word by the key vectors (K) of all words in the sequence: H 0 *W Q *H 0 *W K , to obtain the relevance between words (QK T )

[0020] S202, normalize the relevance (QK T ) through softmax as follows. Let the vector (where z i represents the i-th element in the vector z) calculate the softmax value of each element in the vector z:

[0021] (where K represents the length of the vector z),

[0022] to obtain the normalized vector representation z′ = softmax(z), and then multiply the normalized vector z′ by the value vector V to get the new encoding of each word:

[0023] (where d k is the dimension of the key vector K);

[0024] S203, after concatenating the calculation results of all heads of the multi-head self-attention (Multi-Head Self-Attention), perform a linear transformation through a fully connected layer to reduce the dimension, and use a residual connection to add to the original input, and then perform layer normalization to finally form the output: H 1 = MultiHeadAttention(Q, K, V)+ FeedForward(H 0 );

[0025] S204, use the output H 1 obtained in S203 as the input of S201, and repeatedly execute steps S201 to S203 for L times to calculate the contextualized embedding, and finally use the vector corresponding to the [CLS] token at the beginning of the sentence as the embedding of the entire sentence to obtain the final cut-in vector representation

[0026] Furthermore, the specific method of step S3 is as follows:

[0027] S301, for the set of text embedding vectors X = {x1, x2,... x n}, randomly select k embedding vectors as the initial clustering centroid set ({u1, u2,... u k) This can be done by random selection or a specific strategy. For example, the present invention can divide the text into three parts: front, middle, and back, and then randomly select a vector as the initial centroid.

[0028] S302, Calculate the distance d(x i to each cluster centroid u j using the Euclidean distance for each embedding vector x except the centroid, where d(x i , u j ) = ||x i - u j || (1 ≤ j ≤ k). Let d(x i , u t ) = min_{1≤j≤k} d(x i , u j ), and assign the embedding vector x i to the cluster C t nearest to the centroid;

[0029] S303, For each cluster C j , calculate the average value of all the embedding vectors in the cluster C j . This average value u j becomes the new centroid. The goal of the K-means algorithm is to minimize the sum of the squares of the Euclidean distances from all the embedding vectors to their corresponding cluster centroids, that is, the sum of the squares of the within-cluster errors. The expression of the objective function is as follows:

[0030]

[0031] where K is the number of clusters, C i is the i-th cluster, x is the embedding vector data point in the i-th cluster; u i is the centroid of the i-th cluster, and ‖x - u i ‖ represents the Euclidean distance between the data point x and the cluster centroid u i ;

[0032] S304, Repeatedly execute steps S302 and S303 until the assignment of the vectors to the clusters no longer changes.

[0033] Furthermore, the specific method of step S4 is as follows:

[0034] S401, Select the m vectors C′ j ={x1, x2,... x j} closest to the center in each cluster C j (k = |C m |) as follows: Let C′ j ={xi |x i ∈C j ,d(x i ,u j )=min_{1≤i≤k}d(x i ,u j )} where (u j represents the centroid of the cluster C j ), update the cluster to C j =C j -{x i}, and update k=|C j |-1; repeat the above steps m times, then the vector C′ j ={x1,x2,…x m} can be obtained;

[0035] S402. Using the mathematical idea of permutation and combination, select any vector x j in each cluster C j (x j ∈C′ j ), and then obtain a vector sequence x1x2…x j …x k (1≤j≤k, k represents the number of clusters);

[0036] S403. Arrange the vector sequence x1x2…x j …x k (1≤j≤k, k represents the number of clusters) according to the order of the sentences represented by the vectors in the article to obtain the final embedded vector sequence.

[0037] Furthermore, the specific method of step S5 is as follows:

[0038] S501. Use the BERTSUM encoder to generate the context information embedding vectors of each sentence for the vector H c ={x1,x2,…,x j ,…,x k}(1≤j≤k, k represents the number of clusters) obtained in S4 through the bidirectional Transformer mechanism. Let the encoded vector be h j ,

[0039] H=BERTSUM(H c )={h1,h2,…,h j ,…,h k}(1≤j≤k, k represents the number of clusters)

[0040] H is the vector representation after encoding these sentences;

[0041] S502: The encoded text sequence H = {h1, h2, ..., h k}, and then input to the 6-layer Random Transformer for decoding. Each layer uses the self-attention mechanism to calculate the query vector, key vector, and value vector respectively.

[0042] Q t =W q H ＜t , K t =W k H, V t =W v H

[0043] Where W q , W k , W v is the linear change matrix, H <t is the text sequence generated before time t, and then the query vector (Q t ) respectively and the key vectors of all words in the sequence (K t ) to do the product: W q *H ＜t *W k *H, get the correlation between words Next, calculate the attention weight A t :

[0044]

[0045] Among them, d k is the dimension of the key vector, A t Represents the attention weight of the generated word token at time t;

[0046] S503, based on the attention weight A t , randomly sample the next tokeny from the predicted probability distribution using a multinomial distribution t (where tokeny t is the currently generated token. Token is the basic text unit processed in NLP tasks, representing words):

[0047] y t ~Multinomial(A t )

[0048] The generated word tokeny t Add to the generated sequence and continue decoding at the next moment:

[0049] H ＜t+1 =H <t ∏{y t}

[0050] This process is repeated in the 6 - layer Transformer decoder. The output of each layer affects the decoding result of the next layer until the predetermined generation length is reached. Finally, after 6 - layer decoding, the final generated text Y = {y1, y2, …, y T}. The final output biomedical text summary is as follows:

[0051] Y = RandomTransformer(H).

[0052] Furthermore, it also includes the following steps:

[0053] S6, pairwise candidate summary losses are introduced for all candidate summaries, and margin - based triplet loss is combined to update the weights of the model, so as to optimize the parameters of the encoder and decoder respectively and improve the stability of the generation model.

[0054] Specifically, the specific steps of step S6 are as follows:

[0055] S601, in the configuration, set the target decay rate β1 of the first moment in the Adam optimizer. The target decay rate β1 controls the momentum update of the gradient; set the target decay rate β2 of the second moment. The target decay rate β2 manages the stability of the gradient update; in the warm - up step, adopt the measurement of gradually adding the learning rate to the target decay rate, set the warm - up steps of the encoder to 20000 steps, and the warm - up steps of the decoder to 10000 steps;

[0056] S602, adopt margin - based triplet loss to update the weights of the model:

[0057] Loss1 = max(0, f(D, S) - f(D, S * ) + γ1)

[0058] where D represents the original document, γ1 represents a hyperparameter, S represents the candidate summary, and S * represents the gold summary. This loss function optimizes the model by maximizing the matching score between the original document D and the gold summary S * while minimizing the matching score with other candidate summaries S;

[0059] S603, pairwise candidate summary losses are introduced for all candidate summaries. According to their ROUGE scores relative to the gold summary, all candidate summaries are sorted in descending order:

[0060] Loss2 = max(0, f(D, S i ) - f(D, S j ) + γ2 * (i - j))

[0061] Among them, Loss2 represents the pairwise candidate summary loss based on ROUGE ranking, D represents the original document, and S i represents the i-th ranked candidate summary, and S j represents the j-th ranked candidate summary; γ2 is a hyperparameter used to distinguish high-quality and low-quality candidate summaries;

[0062] S604, the parameters of the encoder and decoder are optimized using the margin-based triple loss, and the final loss function (Loss) is composed of the margin-based triple loss and the pairwise candidate summary loss:

[0063] Loss = Loss1 + Loss2

[0064] Ensures that the gold summary reaches the highest matching score, and better candidate summaries should obtain higher scores than unqualified candidate summaries.

[0065] The present invention adopts the above technical solutions. In biomedical text generation, the PreBeBio algorithm is used, and a new biomedical text generation scheme is proposed. This scheme adopts a new method combining extraction and generation methods to generate biomedical summaries, and ensures a higher semantic similarity between the extracted text and the source text, emphasizing the overall semantic consistency between the candidate summary and the original document; secondly, the present invention combines the context embedding vectors generated by the pre-trained BERT model with the K-Means clustering algorithm to extract biomedical text summaries; BERT provides more powerful semantic representations, enabling it to better capture key sentences in the text, while the K-Means algorithm helps to identify representative sentences in each key sentence cluster; the central sentences effectively capture the essence of their respective clusters, thereby improving the overall quality of the summary. The effective effects of the present invention are as follows: 1. The experimental results on real-world datasets show that compared with some existing unadjusted algorithms, the method of the present invention can achieve more accurate biomedical text generation; 2. The quality of the biomedical text generated by the scheme of the present invention is 0.05% - 3.7% higher than that of the benchmark method, with significant improvements in matching sentence pairs, word pairs, and long sentences; 3. In addition, there is also a 0.14% - 0.67% improvement in the ability of two independent word matching and sentence matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The following further describes the present invention in detail with reference to the drawings and specific embodiments;

[0067] Figure 1 is a schematic flow chart of a biomedical text pre-training generation method in the present invention;

[0068] Figure 2 is a schematic diagram of the final input representation of a sentence in the present invention;

[0069] Figure 3 Schematic diagram of the transformer core of Bert used in the present invention. Detailed implementation manners

[0070] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0071] With the emergence of large-scale models, these models have demonstrated superior language understanding and generation capabilities, which have made it possible to efficiently generate coherent, personalized, and high-quality text. However, directly applying BERT to biomedical text summarization may lead to inaccurate results, problems related to accuracy and reliability, lack of context understanding, and insufficient domain specificity.

[0072] In order to further improve the diversity and accuracy of the generated text, a biomedical text pre-training generation method proposed by the present invention adopts a new method combining extraction and generation methods to generate biomedical summaries, and ensures a higher semantic similarity between the extracted text and the source text, emphasizing the overall semantic consistency between the candidate summary and the original document; secondly, the present invention combines the context embedding vectors generated by the pre-trained BERT model with the K-Means clustering algorithm to extract biomedical text summaries; BERT provides more powerful semantic representations, enabling it to better capture the key sentences in the text, while the K-Means algorithm helps to identify the representative sentences in each key sentence cluster; the central sentences effectively capture the essence of their respective clusters, thereby improving the overall quality of the summary.

[0073] As Figures 1 to 3 shown, the present invention discloses a biomedical text pre-training generation method, which includes the following steps:

[0074] S1. Using the WordPiece tokenization algorithm, divide the input biomedical text sentences into smaller segments or chunks, and then process them through token embeddings, segment embeddings, and position embeddings;

[0075] S2. Process the preprocessed text through the Self-Attention mechanism in the Transformer model to generate text embedding vectors;

[0076] S3. Adopt the K-means clustering algorithm to cluster the text embedding vectors The optimization objective is to minimize the Euclidean distance between the embedding vector and the centroid of its belonging cluster, and iteratively update the centroid until convergence, finally obtaining k clusters, each cluster is represented by a centroid;

[0077] S4. Select the vector closest to the center in each cluster. Each vector represents a sentence in the text. Arrange these sentences in the order they appear in the document.

[0078] S5. Encode the selected context embedding vectors by BertSum and decode them through six layers of RandomTransformer to convert the extracted summary into a generated summary, thus completing the biomedical text generation process.

[0079] Furthermore, the specific method of step S1 is as follows:

[0080] S101. First, as Figure 2 shown, divide the input biomedical text sentences into smaller segments or chunks. For example, divide the two sentences "In this paper" and "We implement biomedical text generation" into three sub-words "In", "this", and "paper", and five sub-words "We", "implement", "biomedical", "text", and "generation". This helps handle out-of-vocabulary words and maintain the efficiency of the model. Add a special [CLS] token at the beginning of each biomedical sentence to represent the information of the whole sentence. For example, add the [CLS] token in front of "In" to represent the information of the whole sentence, and use a [SEP] token after "paper" and "generation" at the end to represent the sentence boundary. Finally, represent the sentence T as a token sequence (t1, t2,... t 11 ).

[0081] S102. Process each sequence (t1, t2,... t 11 ) in sentence T. Convert each sequence into a vector e(t i ) = TokenEmbedding(t i ) through token embedding.

[0082] Then convert it into a vector s(t i ) = SegmentEmbedding(t i ) through segment embedding.

[0083] Finally, assign a position embedding p(t i ) = PositionEmbedding(t i ) to each sequence.

[0084] The representation of the final sequence is: As Figure 2The word "In" shown is converted into a vector e(t2) = TokenEmbedding(t2) = E through token embedding In , and then through segment embedding, the sentence context s(t2) = SegmentEmbedding(t2) = E is added A . Then, position information p(t2) = PositionEmbedding(t2) = E2 is added to the word. Finally, the final embedded vector representation of the word "In" is

[0085] S103. Perform the processing of S101 and S102 on all sentences in the text to obtain the preprocessed vector H of the entire text 0 .

[0086] Furthermore, the specific method of step S2 is as follows

[0087] S201. As Figure 3 shown, a linear transformation is performed on the input sequence vector through three weight matrices to obtain: query vector Q = H 0 *W Q (W Q is the weight matrix of Q), key vector K = H 0 *W K (W K is the weight matrix of K), value vector V = H 0 *W V (W V is the weight matrix of V); then, the query vector (Q) of each word is multiplied by the key vectors (K) of all words in the sequence respectively: the relevance between words (QK T ) is obtained;

[0088] S202. The relevance (QK T ) is normalized through softmax as follows. Let the vector (where z i represents the i-th element in the vector z) calculate the softmax value of each element in the vector z:

[0089] (where K represents the length of the vector z),

[0090] the normalized vector representation z' = softmax(z) is obtained, and then the normalized vector z' is multiplied by the value vector V to obtain the new encoding of each word:

[0091] (where d k is the dimension of the key vector K);

[0092] In S203, after concatenating the calculation results of all heads of the multi-head self-attention, perform a linear transformation through a fully connected layer to reduce the dimension, and use a residual connection to add to the original input, and then perform layer normalization to finally form the output:

[0093] H 1 = MultiHeadAttention(Q, K, V)+ FeedForward(H 0 );

[0094] In S204, use the output H obtained in S203 1 as the input of S201, and repeatedly execute steps S201 to S203 for L times to calculate the contextualized embedding. Finally, use the vector corresponding to the [CLS] token at the beginning of the sentence as the embedding of the entire sentence to obtain the final cut-in vector representation

[0095] Furthermore, the specific method of step S3 is as follows:

[0096] In S301, for the set of embedding vectors of the text X = {x1, x2,... x n}, randomly select k embedding vectors as the initial cluster centroid set ({u1, u2,... u k}). This can be done by random selection or a specific strategy. For example: The present invention can divide the text into 3 parts, namely the front, middle, and back, and then randomly select vectors in each part as the initial centroids.

[0097] In S302, use the Euclidean distance to calculate the distance d(x i to each cluster centroid u j of each embedding vector except the centroid, d(x i , u j ) = ||x i - u j || (1 ≤ j ≤ k). Let d(x i , u t ) = min_{1≤j≤k}d(x i , u j ), and assign the embedding vector x i to the cluster C t nearest to the centroid;

[0098] In S303, for each clustering cluster C j , calculate the average value of all embedding vectors in the clustering C j This average value u This average value u jIt becomes the new centroid. The goal of the K-means algorithm is to minimize the sum of the squares of the Euclidean distances from all the embedded vectors to their corresponding cluster centroids, that is, the sum of the squares of the within-cluster errors. The expression of the objective function is as follows:

[0099]

[0100] where K is the number of clusters, C i is the i-th cluster, x is the embedded vector data point in the i-th cluster; u i is the centroid of the i-th cluster, ‖x - u i ‖ represents the Euclidean distance between the data point x and the cluster centroid u i ;

[0101] S304. Repeatedly execute steps S302 and S303 until the assignment of vectors to clusters no longer changes.

[0102] Furthermore, the specific method of step S4 is as follows:

[0103] S401. In each cluster C j (k = |C j |), select the m vectors C′ j ={x1, x2, … x m} closest to the center. The steps are as follows: Let C′ j ={x i |x i ∈C j , d(x i , u j ) = min_{1≤i≤k} d(x i , u j )}, where (u j represents the centroid of the cluster C j ). Update the cluster to C j = C j - {x j}, and update k = |C j | - 1; Repeat the above steps m times, then the vector C j ′ = {x1, x2, … x m} can be obtained;

[0104] S402. Using the mathematical idea of permutation and combination, select any vector x j (x j ∈C j ′) in each cluster C j ′, and then obtain a vector sequence x1x2…x j …x k (1 ≤ j ≤ k, k represents the number of clusters);

[0105] S403. For the vector sequence x1, x2, …, x j …, x k (1 ≤ j ≤ k, where k represents the number of clusters), arrange them in the order of the sentences represented by the vectors in the article to obtain the final embedded vector sequence.

[0106] Furthermore, the specific method of step S5 is as follows:

[0107] S501. Use the BERTSUM encoder to generate the context information embedded vectors of each sentence for the vector H c = {x1, x2, …, x j , …, x k}(1 ≤ j ≤ k, where k represents the number of clusters) through the bidirectional Transformer mechanism. Let the encoded vector be h j ,

[0108] H = BERTSUM(H c ) = {h1, h2, …, h j , …, h k}(1 ≤ j ≤ k, where k represents the number of clusters)

[0109] H is the vector representation of these sentences after encoding;

[0110] S502. Input the encoded text sequence H = {h1, j2, …, h k} into a 6 - layer Random Transformer for decoding. At each layer, calculate the query vector, key vector, and value vector respectively through the self - attention mechanism

[0111] Q t = W q H ＜t , K t = W k H, V t = W v H

[0112] where W q , W k , W v are linear transformation matrices, and H ＜t is the text sequence generated before the t - th moment. Then, multiply the query vector (Q t ) of each word with the key vectors (K t ) of all words in the sequence: W q * H ＜t * W k * H to obtain the correlation between words Next, calculate the attention weight A t :

[0113]

[0114] Among them, d k is the dimension of the key vector, and A t represents the attention weight for generating the token at the current moment;

[0115] S503. Based on the attention weight A t , randomly sample the next word token y from the predicted probability distribution using the multinomial distribution t (where token y t is the currently generated token, and a token is the basic text unit processed in the NLP task, representing a word):

[0116] y t ~Multinomial(A t )

[0117] Add the generated word token y t to the generated sequence and continue the decoding for the next moment:

[0118] H <t+1 = H ＜t ∪{y t}

[0119] This process will be repeated in the 6 - layer Transformer decoder. The output of each layer will affect the decoding result of the next layer until the predetermined generation length is reached. Finally, after 6 - layer decoding, the final generated text Y = {y1, y2, …, y T} is obtained. The final output biomedical text summary is as follows:

[0120] Y = RandomTransformer(H).

[0121] Furthermore, the present invention also includes: S6. A new fine - tuning strategy is proposed in the experimental configuration to separately handle the optimization of the encoder and the decoder. The specific method of step S6 is as follows:

[0122] S601. In the configuration, the decay rate of the first moment (gradient) is set in the Adam optimizer, denoted as β1 = 0.9, which controls the momentum update of the gradient. The decay rate of the second moment (gradient square) is set to β2 = 0.9 to manage the stability of the gradient update. Start from a relatively low learning rate in the warm - up step and gradually increase to the target rate. Set the warm - up steps of the encoder to 20000 steps and the warm - up steps of the decoder to 10000 steps;

[0123] S602 adopts a triple margin-based loss to update the weights of the model:

[0124] Loss1 = max(0, f(D, S) - f(d, S * ) + γ1)

[0125] where d represents the original document, γ1 represents a hyperparameter, S represents the candidate summary, and S * represents the gold summary. This loss function optimizes the model by maximizing the matching score between the original document d and the gold summary S * while minimizing the matching score with other candidate summaries S;

[0126] S603 introduces pairwise candidate summary losses for all candidate summaries. According to their ROUGE scores relative to the gold summary, the present invention sorts all candidate summaries in descending order:

[0127] Loss2 = max(0, f(D, S i ) - f(D, S j ) + γ2 * (i - j))

[0128] where Loss2 represents the pairwise candidate summary loss based on ROUGE ranking, D represents the original document, S i represents the i-th ranked candidate summary, and S j represents the j-th ranked candidate summary; γ2 is a hyperparameter used to distinguish high-quality and low-quality candidate summaries; finally, the triple margin-based loss optimizes the parameters of the encoder and decoder, and the final loss function (Loss) is composed of the triple margin-based loss and the pairwise candidate summary loss:

[0129] Loss = Loss1 + Loss2

[0130] Ensures that the gold summary achieves the highest matching score, and better candidate summaries should obtain higher scores than unqualified candidate summaries.

[0131] The present invention adopts the above technical solutions. In biomedical text generation, the PreBeBio algorithm is used. The present invention proposes a new biomedical text generation solution. This solution adopts a new method combining extraction and generation methods to generate biomedical abstracts, and ensures a higher semantic similarity between the extracted text and the source text, emphasizing the overall semantic consistency between the candidate abstract and the original document. Secondly, the present invention combines the context embedding vectors generated by the pre-trained BERT model with the K-Means clustering algorithm to extract biomedical text abstracts. BERT provides more powerful semantic representations, enabling it to better capture key sentences in the text, while the K-Means algorithm helps to identify representative sentences in each key sentence cluster. The central sentences effectively capture the essence of their respective clusters, thereby improving the overall quality of the abstract. The beneficial effects of the present invention are as follows: 1. This method ensures a higher semantic similarity between the extracted text and the source text, emphasizing the overall semantic consistency between the candidate abstract and the original document, rather than simply relying on sentence-by-sentence extraction analysis; 2. Experimental results on real-world datasets show that compared with some existing unadjusted algorithms, the quality of the biomedical text generated by the solution of the present invention is 0.05% - 3.7% higher than the benchmark method, with significant improvements in matching sentence pairs, word pairs, and long sentences. In addition, there is also a 0.14% - 0.67% improvement in the ability of two independent word matching and sentence matching; 4. A new fine-tuning strategy is proposed, such as introducing pairwise candidate abstract losses for all candidate abstracts, improving the model stability;

[0132] Obviously, the described embodiments are part of the embodiments of this application, rather than all embodiments. Without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Generally, the components of the embodiments of this application described and illustrated in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of this application claimed, but merely represents the selected embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

Claims

1. A biomedical text pre-training generation method, characterized in that: It includes the following steps: S1. Use the WordPiece tokenization algorithm to divide the input biomedical text sentence into segments or chunks, and then process it through token embedding, segment embedding, and positional embedding to obtain the preprocessed text H 0 ; S2. Generate a text embedding vector for the preprocessed text H 0 through the self-attention mechanism S3. Use the K-means clustering algorithm for the text embedding vectors to perform clustering. The optimization objective is to minimize the Euclidean distance between the embedding vectors and the centroids of the clusters they belong to, and iteratively update the centroids until convergence. Finally, k clusters are obtained, and each cluster is represented by a centroid; S4. Select the m vectors closest to the center in each cluster, where each vector represents a sentence in the text; arrange the selected sentences in the order they appear in the document; S5. Encode the selected context embedding vectors by BertSum and decode them through six layers of Random Transformer to convert the extracted summary into a generated summary, thus completing the biomedical text generation process.

2. A biomedical text pre-training generation method according to claim 1, characterized in that: The specific steps of step S1 are: S101, Split the input biomedical text sentence into segments or chunks, add a special [CLS] token at the beginning of each biomedical sentence to represent the information of the whole sentence, use a [SEP] token at the end to represent the boundary of the sentence, and finally represent the sentence T as a token sequence (t1, t2, … t n ); S102. Process each sequence (t1, t2, … t n ) in sentence T. Convert each sequence into a vector e(t i ) = TokenEmbedding(t i ) through token embedding, and then convert it into a vector s(t i ) = SegmentEmbedding(t i ) through segment embedding. Finally, assign a position embedding p(t i ) = PositionEmbedding(t i ) to each sequence. The representation of the final sequence of the current sentence is: S103, repeat the processing of S101 and S102 for all sentences in the text to obtain the preprocessed vector H of the entire text 0 。 3. A biomedical text pre-training generation method according to claim 1, characterized in that: The specific steps of step S2 are: S201, perform a linear transformation on the input sequence vector through three weight matrices to obtain: query vector Q = H 0 *W Q , where W Q is the weight matrix of Q; key vector K = H 0 *W K , where W K is the weight matrix of K; value vector V = H 0 *W V , where W V is the weight matrix of V; then multiply the query vector Q of each word with the key vectors K of all words in the sequence to obtain the correlation QK between words T ; S202, take the relevance QK T Then normalize it through softmax; let the vector where z i represents the i-th element in the vector z; Calculate the softmax value of each element in the vector z, where K represents the length of the vector z; obtain the normalized vector representation z' = softmax(z), and then multiply the normalized vector z' by the value vector V to get the new encoding of each word: where d k is the dimension of the key vector K; S203. After concatenating the calculation results of all heads of the multi-head self-attention, perform a linear transformation through a fully connected layer to reduce the dimension, and use a residual connection to add to the original input, and then perform layer normalization to finally form the output H 1 : H 1 = MultiHeadAttention(Q, K, V) + FeedForward(H 0 ); S204, output H 1 As the input of S201, repeat steps S201 to S203 for L times to calculate the contextual embedding. Finally, use the vector corresponding to the [CLS] token at the beginning of the sentence as the embedding of the entire sentence to obtain the final cut-in vector representation 4. A biomedical text pre-training generation method according to claim 1, characterized in that: The preprocessed text in S2 is processed by the Self-Attention mechanism in the Transformer model to generate text embedding vectors 5. A biomedical text pre-training generation method according to claim 1, characterized in that: The K-means clustering algorithm is adopted in S3.

6. A biomedical text pre-training generation method according to claim 1, characterized in that: The specific steps of step S3 are: S301. For the text embedding vector set X = {x1, x2, …, x n}, randomly select k embedding vectors as the initial cluster centroid set ({u1, u2, …, u k}). S302, calculate the Euclidean distance between each embedding vector x (excluding the centroid) i and each cluster centroid u j to obtain the distance d(x i , u j ) = ||x i - u j || (1 ≤ j ≤ k). Let d(x i , u t ) = min_{1≤j≤k} d(x i , u j ), and assign the embedding vector x i to the cluster C that is closest to the centroid t ; S303. For each clustering cluster C j , calculate the average value of all the embedding vectors in the cluster C j as the new centroid; ​ S304. Repeatedly execute steps S302 and S303 until the assignment of vectors to clusters no longer changes.

7. A biomedical text pre-training generation method according to claim 1, characterized in that: The specific steps of step S4 are: S401. In each cluster C j (k = |C j |), select the m vectors C′ j = {x1, x2, … x m} as follows: Let C′ j = {x i |x i ∈ C j , d(x i , u j ) = min1≤i≤k d(x i , u j )}, where u j represents the centroid of cluster C j ; Update the cluster to C j = C j - {x i}, and update k = |C j | - 1; Repeat the above steps m times, then obtain the vector C′ j = {x1, x2, … x m}; S402, using the mathematical idea of permutation and combination, select any vector x j from each clustering cluster C j , and x j ∈C′ j ; furthermore, obtain a vector sequence x1x2…x j …x k , 1 ≤ j ≤ k, where k represents the number of clusters; S403. For the vector sequence x1x2…x j …x k , where 1≤h≤k and k represents the number of clusters; arrange them in the order of the sentences represented by the vectors in the article to obtain the final embedded vector sequence.

8. A biomedical text pre-training generation method according to claim 7, characterized in that: The specific steps of step S5 are: S501, use the BERT SUM encoder to obtain the vector H obtained in S4 c ={x1, x2, …, x j , …, x k}, 1 ≤ j ≤ k, where k represents the number of clusters; generate the context information embedding vector of each sentence through the bidirectional Transformer mechanism; the specific expression is as follows: H = BERTSUM(H c ) = {h1, h2, …, h j , …, h k}, 1 ≤ j ≤ k, where k represents the number of clusters; Among them, h j is the encoded vector, and H is the vector representation after sentence encoding; S502. Input the encoded text sequence H = {h1, h2, …, h k} into a 6-layer Random Transformer for decoding. At each layer, calculate the query vector, key vector, and value vector respectively through the self-attention mechanism Q t = W q H ＜t , K t = W k H, V t = W v H Among them, W q , W k , W v is a linear transformation matrix, H ＜t is the text sequence generated before time t. Next, calculate the attention weight A t : Among them, d k is the dimension of the key vector, and A t represents the attention weight for generating the token at time t; S503, based on the attention weight A t , randomly sample the next word token y from the predicted probability distribution using the multinomial distribution t , and the corresponding expression is: y t ~Multinomial(A t ); S504, add the generated tokeny t to the generated sequence and continue decoding for the next time step: H ＜t+1 = H ＜t ∪ {y t}; S505 is repeated in the 6-layer Transformer decoder, and the output of each layer affects the decoding result of the next layer until the predetermined generation length is reached; after 6-layer decoding, the final generated text Y = {y1, y2, …, y T}; the final output biomedical text summary is as follows: Y = RandomTransformer(H).

9. A biomedical text pre-training generation method according to claim 1, characterized in that: It also includes the following steps: S6. Introduce pairwise candidate summary losses for all candidate summaries and combine margin-based triplet losses to update the weights of the model, so as to optimize the parameters of the encoder and decoder respectively and improve the stability of the generation model.

10. A biomedical text pre-training generation method according to claim 9, characterized in that: The specific steps of step S6 are: S601. In the configuration, set the target decay rate β1 of the first moment in the Adam optimizer, and the target decay rate β1 controls the momentum update of the gradient; set the target decay rate β2 of the second moment, and the target decay rate β2 manages the stability of the gradient update; adopt a measurement of gradually adding the learning rate to the target decay rate in the warm-up step, set the warm-up steps of the encoder to 20,000 steps, and the warm-up steps of the decoder to 10,000 steps; S602. Adopt margin-based triplet losses to update the weights of the model: Loss1 = max(0, f(D, S) - f(D, S * ) + γ1) Among them, D represents the original document, γ1 represents a hyperparameter, and S represents the candidate summary; S * represents the gold summary; this loss function optimizes the model by maximizing the matching score between the original document D and the gold summary S * while minimizing the matching score with other candidate summaries S; S603. Introduce pairwise candidate summary losses for all candidate summaries, and sort all candidate summaries in descending order according to the ROUGE score relative to the gold summary: Loss2 = max(0, f(D, S i ) - f(D, S j ) + γ2 * (i - j)) Among them, Loss2 represents the pairwise candidate summary loss based on ROUGE ranking, D represents the original document, and S i represents the i-th ranked candidate summary, and S j represents the j-th ranked candidate summary; γ2 is a hyperparameter used to distinguish high-quality and low-quality candidate summaries; S604. Optimize the parameters of the encoder and decoder based on margin-based triplet losses. The final loss function Loss is jointly composed of margin-based triplet losses and pairwise candidate summary losses: Loss = Loss1 + Loss2.

Images