Medical literature evaluation method, electronic device, storage medium and program product

Through a quality evaluation model based on sentence vector similarity, the twin network structure and biomedical pre-training model are used to evaluate the quality of medical literature, which solves the problems of low evaluation efficiency and poor accuracy in the existing technology, and achieves efficient and accurate Evaluation of medical literature quality.

CN114048729BActive Publication Date: 2024-11-19SUN YAT SEN MEMORIAL HOSPITAL SUN YAT SEN UNIV +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202111216927.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-19
Publication Date
2024-11-19
Estimated Expiration
2041-10-19

AI Technical Summary

Technical Problem

The prior art does not use NLP technology to evaluate the quality of medical literature, resulting in low evaluation efficiency and poor accuracy.

Method used

The quality evaluation model based on sentence vector similarity is adopted, and the quality evaluation model of medical literature is evaluated using twin network structure and biomedical pre-trained model. The quality evaluation model with high accuracy and recall is obtained through feature extraction and similarity calculation training.

Benefits of technology

It has achieved efficient and accurate quality evaluation of medical literature through artificial intelligence technology to meet the needs of medical literature quality evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114048729B_ABST
    Figure CN114048729B_ABST
Patent Text Reader

Abstract

The present invention provides a medical literature evaluation method, electronic device, storage medium and program product, the method comprising obtaining a text to be evaluated of a medical literature to be evaluated; inputting the text to be evaluated into a quality evaluation model, performing quality evaluation, and obtaining a quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is trained based on sentence vector similarity results, wherein the sentence vector similarity results are obtained by performing similarity calculation on sentence vectors generated by a sentence vector generation model of a twin network structure, wherein the sentence vector generation model comprises a biomedical pre-training model. The present invention implements model training based on similar sentence vectors, and then trains to obtain a quality evaluation model with high precision and high recall, thereby meeting the needs of medical literature quality evaluation, and ultimately achieving quality evaluation of medical literature through artificial intelligence technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of text analysis, and in particular to a medical literature evaluation method, electronic equipment, storage medium and program product. Background Art

[0002] NLP (natural language processing) is an important direction in the fields of computer science and artificial intelligence. It includes various theories and methods for effective communication between humans and computers using natural language.

[0003] With the development of NLP technology, especially after the release of the BERT (Bidirectional Encoder Representation from Transformers) pre-training model, text tasks based on BERT have higher quality and efficiency. In the biomedical field, due to the particularity of biomedical text data, ordinary text mining methods cannot handle these medical terms well. Based on this, a biomedical pre-training model (PubMedBERT) is proposed to achieve text tasks in the biomedical field.

[0004] However, the existing technology does not use NLP technology to evaluate the quality of medical literature, let alone use biomedical pre-training models to evaluate the quality of medical literature. If NLP technology is not used to evaluate the quality of medical literature, it will lead to low evaluation efficiency and poor accuracy. Therefore, how to use artificial intelligence technology to evaluate the quality of medical literature is an urgent problem to be solved. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a medical literature evaluation method, electronic equipment, storage medium and program product.

[0006] The present invention provides a medical literature evaluation method, comprising:

[0007] Obtain the text to be reviewed of the medical literature to be reviewed;

[0008] The text to be evaluated is input into a quality evaluation model for quality evaluation, and a quality evaluation result output by the quality evaluation model is obtained. The quality evaluation model is trained based on sentence vector similarity results. The sentence vector similarity results are obtained by calculating the similarity of sentence vectors generated by a sentence vector generation model of a twin network structure. The sentence vector generation model includes a biomedical pre-training model.

[0009] A medical literature evaluation method provided by the present invention also includes a training method for the quality evaluation model:

[0010] Acquire a first sentence text and a second sentence text as training data, input the first sentence text and the second sentence text into a sentence vector generation model of a twin network structure, perform feature extraction, and obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, wherein the sentence vector generation model includes a biomedical pre-training model;

[0011] Calculate the similarity between the first sentence vector and the second sentence vector to obtain a sentence vector similarity result;

[0012] Based on the sentence vector similarity results, the model to be trained is trained to obtain a quality evaluation model.

[0013] According to a medical literature evaluation method provided by the present invention, the sentence vector generation model includes a first sentence vector generation sub-model and a second sentence vector generation sub-model, and the first sentence text and the second sentence text are input into the sentence vector generation model of the twin network structure, and feature extraction is performed to obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, including:

[0014] Inputting the first sentence text into the first sentence vector generation sub-model, performing feature extraction, and obtaining a first sentence vector output by the first sentence vector generation sub-model;

[0015] The second sentence text is input into the second sentence vector generation sub-model to perform feature extraction to obtain a second sentence vector output by the second sentence vector generation sub-model.

[0016] According to a medical literature evaluation method provided by the present invention, the first sentence vector generation sub-model includes a biomedical pre-training model and a pooling layer, and the first sentence text is input into the first sentence vector generation sub-model to perform feature extraction to obtain a first sentence vector output by the first sentence vector generation sub-model, including:

[0017] Inputting the first sentence text into the biomedical pre-training model, performing feature extraction, and obtaining a first feature vector output by the biomedical pre-training model;

[0018] The first feature vector is input into the pooling layer for pooling processing to obtain a first sentence vector output by the pooling layer.

[0019] According to a medical literature evaluation method provided by the present invention, the first feature vector is a first word vector, and the first feature vector is input into the pooling layer for pooling processing to obtain a first sentence vector output by the pooling layer, including:

[0020] The first word vector is input into the pooling layer, and average pooling is performed to obtain the first sentence vector output by the pooling layer.

[0021] According to a medical literature evaluation method provided by the present invention, before training the model to be trained to obtain the quality evaluation model based on the sentence vector similarity result, the method further includes:

[0022] Obtaining a distance vector between the first sentence vector and the second sentence vector;

[0023] Concatenate the first sentence vector, the second sentence vector and the distance vector to obtain an aggregated vector;

[0024] The step of training the model to be trained based on the sentence vector similarity result to obtain a quality evaluation model includes:

[0025] Based on the sentence vector similarity result and the aggregated vector, the model to be trained is trained to obtain a quality evaluation model.

[0026] According to a medical literature evaluation method provided by the present invention, the model to be trained includes a dropout layer, and the quality evaluation model is obtained by training the model to be trained, including:

[0027] Based on the Dropout, two output distributions of the same input are obtained;

[0028] Calculating the KL divergence and the cross entropy loss of the two output distributions, and calculating the total loss of the KL divergence and the cross entropy loss;

[0029] Based on the total loss, the model to be trained is trained to obtain a quality evaluation model.

[0030] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned medical literature evaluation methods are implemented.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the medical literature evaluation methods described above.

[0032] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned medical literature evaluation methods are implemented.

[0033] The medical literature evaluation method, electronic device, storage medium and program product provided by the present invention obtain the text to be evaluated of the medical literature to be evaluated, and then input the text to be evaluated into the quality evaluation model to perform quality evaluation, and obtain the quality evaluation result output by the quality evaluation model, thereby realizing the quality evaluation of medical literature through artificial intelligence technology. At the same time, since the quality evaluation model is trained based on the sentence vector similarity result, the sentence vector similarity result is obtained by calculating the similarity of the sentence vector generated by the sentence vector generation model of the twin network structure, thereby realizing model training based on similar sentence vectors, and then training to obtain a quality evaluation model with high precision and high recall rate, thereby meeting the needs of medical literature quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 One of the flow charts of the medical literature evaluation method provided by the present invention;

[0036] Figure 2 The second flowchart of the medical literature evaluation method provided by the present invention;

[0037] Figure 3 Flow chart 3 of the medical literature evaluation method provided by the present invention;

[0038] Figure 4 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Figure 1 One of the flow charts of the medical literature evaluation method provided by the present invention is as follows: Figure 1 As shown, the medical literature evaluation method provided by the present invention comprises:

[0041] Step 110, obtaining the text to be evaluated of the medical document to be evaluated;

[0042] In this embodiment, the medical document to be evaluated is a medical document that needs to be evaluated for quality. The medical document to be evaluated may be a target medical document selected by a user, or may be a medical document automatically selected by an application device of the medical document evaluation method. The medical document to be evaluated may be a document in an existing document library involving the biological or medical field.

[0043] The text to be evaluated includes at least one of the following: abstract, document title, journal, impact factor, background content or first paragraph of text, etc.

[0044] In one embodiment, the step 110 includes:

[0045] Get abstracts of medical literature to be reviewed.

[0046] Step 120, input the text to be evaluated into a quality evaluation model, perform quality evaluation, and obtain a quality evaluation result output by the quality evaluation model. The quality evaluation model is trained based on the sentence vector similarity result. The sentence vector similarity result is obtained by calculating the similarity of the sentence vector generated by the sentence vector generation model of the twin network structure. The sentence vector generation model includes a biomedical pre-training model.

[0047] In this embodiment, the types of quality evaluation results can be set according to actual needs, for example, divided into three types of quality evaluation results: high, medium, and low. Of course, it can also be classified into two or more types, which is not limited here.

[0048] Among them, the biomedical pre-training model (PubMedBERT) is used to generate semantically meaningful embeddings for subsequent fixed-length sentence vectors. That is, the sentence vector has semantic meaning. Based on this, the sentence vector is similarly calculated to obtain the semantic similarity distance, so that semantically similar sentences are used as positive samples for model training. In other words, the quality evaluation model is trained based on the sentence vector similarity results.

[0049] The quality evaluation model is a machine learning model. Specifically, the quality evaluation model is a model that uses a natural language processing (NLP) algorithm to perform text classification tasks.

[0050] In a specific embodiment, the quality evaluation model includes a feature extractor and a classifier. Specifically, based on the feature extractor in the trained quality evaluation model, the text feature information of the text to be evaluated is extracted, and then based on the text feature information and the classifier in the quality evaluation model, the text feature information is classified and predicted to obtain a classification prediction result, that is, a quality evaluation result output by the quality evaluation model.

[0051] Among them, the feature extractor can generate a model for sentence vectors, thereby extracting the sentence vectors of the text to be evaluated, and then classifying and predicting the sentence vectors based on the sentence vectors and the classifier in the quality evaluation model to obtain the classification prediction results.

[0052] The specific execution process of the classifier is to obtain a classification probability vector (i.e., the possibility prediction of the input text to be evaluated for each category), and then determine the quality evaluation result corresponding to the maximum classification probability value in the classification probability vector. The classifier may include a fully connected layer and a softmax function.

[0053] In some embodiments, the feature extractor is an encoder, the encoder includes a sentence vector generation model, the sentence vector generation model includes a fine-tuned biomedical pre-trained model and a pooling layer, the classifier includes a fully connected layer and a softmax function, and the above step 120 includes:

[0054] Based on the encoder of the quality evaluation model, feature extraction is performed on the text to be evaluated to obtain a text feature vector; based on the fully connected layer of the quality evaluation model, category prediction is performed on the text feature vector to obtain a quality evaluation result output by the quality evaluation model.

[0055] Among them, the specific execution process of the fully connected layer is to obtain the classification probability vector (that is, the possibility prediction of the input text to be evaluated for each category) through the softmax function, and then determine the quality evaluation result corresponding to the largest classification probability value in the classification probability vector.

[0056] Among them, the fine-tuned biomedical pre-training model is used to generate a token vector, which can be a word vector (word embedding vector); of course, if other segmentation strategies (word segmentation strategies) are adopted, the token vector is a segmentation feature vector.

[0057] The pooling layer is used to average or max pool all token vectors to obtain the corresponding sentence vector. For example, by averaging all token dimensions of a sentence through Pooling, a vector of 768 dimensions can be obtained. In addition, pooling can greatly reduce network parameters and avoid overfitting.

[0058] In order to train the quality evaluation model, the method for training the quality evaluation model is also included:

[0059] Acquire text data, where the text data is data in a document library; annotate the text data with labels for quality evaluation results to obtain quality evaluation result label data; acquire a model to be trained, and select training sample data from the text data and the quality evaluation result label data; iteratively train the model to be trained based on the training sample data and a preset loss function to obtain the quality evaluation model.

[0060] Specifically, training samples are selected from the training sample data, and the text data and quality evaluation result label data corresponding to the training samples are input into the model to be trained, model prediction is performed, and the model output label is obtained, and then based on the preset loss function, the model loss is calculated, and then based on the model loss, the model to be trained is updated until the number of iterations of the model to be trained reaches the preset number of iterations or the corresponding loss function (objective function) reaches the preset value.

[0061] It should be noted that the appropriate number of iterations can be adjusted continuously based on the training effect. In addition, through gradient descent, the optimal weight value that minimizes the objective function can be found, and the weight value can be learned autonomously through training, thereby updating the model to be trained.

[0062] In addition, it should be noted that the training data is mainly annotated for RCT and Meta based on the existing literature library data. The annotation process mainly relies on professional physicians to manually annotate the training data according to the mainstream medical literature evaluation system AMSTAR1, AMSTAR2, GRADE, etc. to obtain training sample data.

[0063] Furthermore, the training sample data is divided into a training set and a validation set. For example, the training sample data is divided into a training set and a validation set according to a certain ratio. The training set is used to train the model, and the validation set is used to validate the model. Specifically, the training set is used for training so that the objective function is as small as possible, and the validation set is used to evaluate the validation model after each round of training until the model converges and the weight of the model is derived, thereby obtaining the final quality evaluation model.

[0064] Among them, since the training sample data includes positive samples and negative samples, before the model to be trained is iteratively trained based on the training sample data and the preset loss function to obtain the quality evaluation model, positive samples are first screened from the training sample data, and then the model to be trained is iteratively trained based on the positive samples and the preset loss function to obtain the quality evaluation model.

[0065] Specifically, the screening method of positive samples is as follows: obtain the first sentence text and the second sentence text as training data, input the first sentence text and the second sentence text into the sentence vector generation model of the twin network structure, perform feature extraction, obtain the first sentence vector corresponding to the first sentence text and the second sentence vector corresponding to the second sentence text, and the sentence vector generation model includes a biomedical pre-training model; perform similarity calculation on the first sentence vector and the second sentence vector to obtain a sentence vector similarity result; determine the positive samples in the training data based on the sentence vector similarity result; and train the model to be trained based on the positive samples to obtain a quality evaluation model. In other words, use the similarity algorithm to obtain sentences with similar semantics, so as to perform model training based on sentences with similar semantics (positive samples).

[0066] It can be understood that quality evaluation of medical literature is carried out to obtain quality evaluation results so that the quality evaluation results can better serve scientific research, platform building fields, etc. such as evidence-based medicine research, etc.

[0067] According to the medical literature evaluation method of the embodiment of the present invention, by obtaining the text to be evaluated of the medical literature to be evaluated, and then inputting the text to be evaluated into the quality evaluation model, quality evaluation is performed, and the quality evaluation result output by the quality evaluation model is obtained, thereby realizing the quality evaluation of medical literature through artificial intelligence technology. At the same time, since the quality evaluation model is trained based on the sentence vector similarity result, the sentence vector similarity result is obtained by calculating the similarity of the sentence vector generated by the sentence vector generation model of the twin network structure, thereby realizing model training based on similar sentence vectors, and then training to obtain a quality evaluation model with high precision and high recall rate, thereby meeting the needs of medical literature quality evaluation.

[0068] Furthermore, based on the above-mentioned first embodiment, a second embodiment of the medical literature evaluation method of the present invention is proposed. Figure 2 The second flowchart of the medical literature evaluation method provided by the present invention is as follows: Figure 2 As shown, in this embodiment, a training method of the quality evaluation model is also included:

[0069] Step 210, obtaining a first sentence text and a second sentence text as training data, inputting the first sentence text and the second sentence text into a sentence vector generation model of a twin network structure, performing feature extraction, and obtaining a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, wherein the sentence vector generation model includes a biomedical pre-training model;

[0070] In this embodiment, the first sentence text includes one sentence, and the second sentence text includes one sentence. The first sentence text and the second sentence text are two sentences obtained based on the text data to be trained.

[0071] The sentence vector generation model of the twin network structure includes two sub-models, which are two sub-networks with the same structure and shared weights. During the training process, since the weights of the two sub-models are shared, the computational overhead can be saved, thereby improving the training efficiency.

[0072] Among them, the biomedical pre-training model (PubMedBERT) is used to generate embeddings with semantic meaning for subsequent fixed-length sentence vectors. That is, the first sentence vector and the second sentence vector have semantic meaning, so that the first sentence vector and the second sentence vector can be used for similarity calculation to obtain the semantic similarity distance, so that the semantically similar first sentence text and the second sentence text are used as positive samples for model training. In other words, the quality evaluation model is trained based on the sentence vector similarity results.

[0073] Specifically, the first sentence text and the second sentence text are input into a sentence vector generation model of a twin network structure, and semantic features are extracted to obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text. The sentence vector generation model includes a biomedical pre-training model for extracting semantic meaning, and the first sentence vector and the second sentence vector both include semantic meaning.

[0074] In one embodiment, the sentence vector generation model includes a first sentence vector generation sub-model and a second sentence vector generation sub-model, and the above step 210 includes:

[0075] Step 211, inputting the first sentence text into the first sentence vector generation sub-model, performing feature extraction, and obtaining a first sentence vector output by the first sentence vector generation sub-model;

[0076] Step 212: input the second sentence text into the second sentence vector generation sub-model, perform feature extraction, and obtain the second sentence vector output by the second sentence vector generation sub-model.

[0077] In this embodiment, the first sentence vector generation sub-model and the second sentence vector generation sub-model have the same network structure and share weights.

[0078] Since the weights of the first sentence vector generation sub-model and the second sentence vector generation sub-model are shared, the computational overhead can be saved during the training process, thereby improving the training efficiency of the model.

[0079] It should be noted that both the first sentence vector generation sub-model and the second sentence vector generation sub-model include a biomedical pre-training model (PubMedBERT), which can generate a first sentence vector with semantic meaning and a second sentence vector with semantic meaning, so as to facilitate subsequent similarity calculation of the first sentence vector and the second sentence vector to obtain a semantic similarity distance, thereby using the semantically similar first sentence text and the second sentence text as positive samples for model training.

[0080] In another embodiment, the first sentence vector generation sub-model includes a biomedical pre-training model and a pooling layer, and the above step 211 includes:

[0081] Step 2111, inputting the first sentence text into the biomedical pre-training model, performing feature extraction, and obtaining a first feature vector output by the biomedical pre-training model;

[0082] In this embodiment, the biomedical pre-training model is used to generate a token vector, which can be a word vector (word embedding vector); of course, if other segmentation strategies (word segmentation strategies) are adopted, the token vector is a segment feature vector. Based on this, the word vector and the segment feature vector are collectively referred to as feature vectors.

[0083] In one embodiment, the above step 2111 includes:

[0084] The first sentence text is input into the biomedical pre-training model to perform feature extraction to obtain a word vector output by the biomedical pre-training model.

[0085] In another embodiment, the step 2111 includes:

[0086] The first sentence text is input into the biomedical pre-training model to perform feature extraction to obtain a segmented feature vector output by the biomedical pre-training model.

[0087] Step 2112: input the first feature vector to the pooling layer, perform pooling processing, and obtain the first sentence vector output by the pooling layer.

[0088] The pooling layer is used to perform average pooling or maximum pooling on all first feature vectors to output the corresponding first sentence vector. Pooling can greatly reduce network parameters and avoid overfitting.

[0089] In one embodiment, the first feature vector is input into the pooling layer, and average pooling processing is performed to obtain the first sentence vector output by the pooling layer.

[0090] For example, by averaging all token dimensions of a sentence through Pooling, a vector of 768 dimensions can be obtained.

[0091] In another embodiment, the first feature vector is input into the pooling layer, and a maximum pooling process is performed to obtain a first sentence vector output by the pooling layer.

[0092] In another embodiment, the first feature vector is a first word vector, and the step 2112 includes:

[0093] Step 21121, input the first word vector into the pooling layer, perform average pooling processing, and obtain the first sentence vector output by the pooling layer.

[0094] In this embodiment, the first word vector is also referred to as a first word embedding vector. Performing average pooling can obtain important information in the first word vector, and then generate a first sentence vector based on the important information.

[0095] In another embodiment, the second sentence vector generation sub-model includes a biomedical pre-training model and a pooling layer, and the above step 212 includes:

[0096] The second sentence text is input into the biomedical pre-training model to perform feature extraction to obtain a second feature vector output by the biomedical pre-training model; the second feature vector is input into the pooling layer to perform pooling processing to obtain a second sentence vector output by the pooling layer.

[0097] Among them, the biomedical pre-training model is used to generate a token vector, which can be a word vector (word embedding vector); of course, if other segmentation strategies (word segmentation strategies) are adopted, the token vector is a segment feature vector. Based on this, the word vector and the segment feature vector are collectively referred to as feature vectors.

[0098] In one embodiment, the second sentence text is input into the biomedical pre-training model, feature extraction is performed, and a word vector output by the biomedical pre-training model is obtained.

[0099] In another embodiment, the second sentence text is input into the biomedical pre-training model to perform feature extraction to obtain a segmented feature vector output by the biomedical pre-training model.

[0100] The pooling layer is used to perform average pooling or maximum pooling on all second feature vectors to output the corresponding second sentence vector. Pooling can greatly reduce network parameters and avoid overfitting.

[0101] In one embodiment, the second feature vector is input to the pooling layer, and average pooling is performed to obtain a second sentence vector output by the pooling layer. For example, by calculating the average of all token dimensions of the sentence through Pooling, a vector of 768 dimensions can be obtained.

[0102] In another embodiment, the second feature vector is input into the pooling layer, and a maximum pooling process is performed to obtain a second sentence vector output by the pooling layer.

[0103] In another embodiment, the second feature vector is a second word vector, and the second word vector is input into the pooling layer, and average pooling is performed to obtain a second sentence vector output by the pooling layer. The second word vector is also called a second word embedding vector. Average pooling can obtain important information in the second word vector, and then generate a second sentence vector based on the important information.

[0104] In addition, it should be noted that the training data can be based on existing literature library data, with training data annotation mainly for two types of RCT and Meta. The annotation process mainly relies on professional physicians to manually annotate the training data based on mainstream medical literature evaluation systems such as AMSTAR1, AMSTAR2, GRADE, etc. to obtain training data.

[0105] Further, the training data is divided into a training set and a validation set, for example, the training data is divided into a training set and a validation set according to a certain ratio, wherein the training set is used to train the model, and the validation set is used to validate the model.

[0106] Step 220, calculating the similarity between the first sentence vector and the second sentence vector to obtain a sentence vector similarity result;

[0107] In this embodiment, the sentence vector similarity result is a semantic similarity result, that is, both the first sentence vector and the second sentence vector have semantic meanings, and the semantic similarity is calculated for the two.

[0108] Among them, similarity calculation methods may include cosine similarity, Manhattan distance calculation, and Euclidean distance calculation.

[0109] In one embodiment, the step 220 includes:

[0110] The first sentence vector and the second sentence vector are subjected to cosine similarity calculation to obtain a sentence vector similarity result. Specifically, the first sentence vector and the second sentence vector are subjected to cosine similarity calculation to obtain a sentence vector cosine distance; and the sentence vector similarity result is determined based on the sentence vector cosine distance and a preset distance.

[0111] In another embodiment, the step 220 includes:

[0112] The first sentence vector and the second sentence vector are subjected to Euclidean distance calculation to obtain a sentence vector similarity result. Specifically, the first sentence vector and the second sentence vector are subjected to Euclidean distance calculation to obtain a sentence vector Euclidean distance; and the sentence vector similarity result is determined based on the sentence vector Euclidean distance and a preset distance.

[0113] It should be noted that the similarity distance obtained by similarity calculation is compared with the preset distance. If the similarity distance is less than the preset distance, it is determined to be a positive sample, that is, it is determined to be a sentence with similar semantics; if the similarity distance is greater than or equal to the preset distance, it is determined to be a negative sample.

[0114] Step 230: Based on the sentence vector similarity result, the model to be trained is trained to obtain a quality evaluation model.

[0115] In this embodiment, since the training data includes positive samples and negative samples, before model training, positive samples are first screened from the training data, and then the training model is iteratively trained based on the positive samples and a preset loss function to obtain a quality evaluation model.

[0116] Specifically, based on the sentence vector similarity results, positive samples in the training data are determined; based on the positive samples, the training model is trained to obtain a quality evaluation model. In other words, sentences with similar semantics are obtained using a similarity algorithm, and model training is performed based on sentences with similar semantics (positive samples).

[0117] In one embodiment, the quality evaluation model obtained through training can be evaluated using F-Measure. The formula of F-Measure is:

[0118]

[0119] Among them, precision P represents the proportion of correct samples among all predicted positive samples. Recall R represents the proportion of all positive samples that are predicted to be positive samples. Generally speaking, precision and recall restrict each other, so F-Measure is used to comprehensively evaluate the model. The higher the F-Measure, the better the performance.

[0120] In another embodiment, the quality evaluation model obtained through training can be evaluated using accuracy, where Accuracy represents the ratio of the number of samples correctly classified by the model to the total number of samples for a given data set.

[0121] In this embodiment, by obtaining the first sentence text and the second sentence text as training data, the first sentence text and the second sentence text are input into the sentence vector generation model of the twin network structure, and feature extraction is performed to obtain the first sentence vector corresponding to the first sentence text and the second sentence vector corresponding to the second sentence text, thereby performing similarity calculation on the first sentence vector and the second sentence vector to obtain the sentence vector similarity result, and then realizing model training based on similar sentence vectors, and then training to obtain a quality evaluation model with high precision and high recall rate, that is, training to obtain a high-performance quality evaluation model, and ultimately improving the accuracy of medical literature quality evaluation.

[0122] Furthermore, based on the above second embodiment, a third embodiment of the medical literature evaluation method of the present invention is proposed. Figure 3 The third flowchart of the medical literature evaluation method provided by the present invention is as follows: Figure 3 As shown, in this embodiment, before the above step 230, the medical literature evaluation method further includes:

[0123] Step 310, obtaining a distance vector between the first sentence vector and the second sentence vector;

[0124] In this embodiment, the distance vector between the first sentence vector and the second sentence vector may be a vector obtained by performing similarity calculation on the above step 220, or may be obtained by performing similarity calculation on the first sentence vector and the second sentence vector again.

[0125] Among them, similarity calculation methods may include cosine similarity, Manhattan distance calculation, and Euclidean distance calculation.

[0126] In one embodiment, based on the to-be-trained model of the three-level network structure, a distance vector between the first sentence vector and the second sentence vector is obtained.

[0127] For ease of understanding, assume that the first sentence vector is u and the second sentence vector is v, then the distance vector is |uv|.

[0128] Step 320, concatenating the first sentence vector, the second sentence vector and the distance vector to obtain an aggregate vector;

[0129] In this embodiment, the dimension of the aggregated vector is three times the dimension of the first sentence vector, the dimension of the aggregated vector is three times the dimension of the second sentence vector, and the dimension of the aggregated vector is three times the dimension of the distance vector.

[0130] For ease of understanding, assume that the first sentence vector is u and the second sentence vector is v. Use the feature extraction strategy of (u, v, |uv|), that is, concatenate the three vectors u, v, and |uv|, and finally obtain an aggregate vector with a dimension of 3*d (d is the dimension of the first sentence vector).

[0131] Accordingly, the above step 230 includes:

[0132] Step 231: Based on the sentence vector similarity result and the aggregated vector, the model to be trained is trained to obtain a quality evaluation model.

[0133] In this embodiment, the first sentence vector, the second sentence vector and the distance vector therebetween are concatenated to obtain an aggregate vector, thereby performing model training based on a higher-dimensional aggregate vector. Compared with the prior art of using low-dimensional sentence vectors for model training, this embodiment can improve the model training effect to obtain a quality evaluation model with higher accuracy, thereby improving the accuracy of medical literature quality evaluation.

[0134] Further, based on the above embodiments, a fourth embodiment of the medical literature evaluation method of the present invention is proposed. In this embodiment, the model to be trained includes a dropout layer. In the above step 230, the model to be trained is trained to obtain a quality evaluation model, including:

[0135] Step 232, based on the Dropout, obtain two output distributions of the same input;

[0136] In this embodiment, in order to avoid overfitting of model training, Dropout (discard layer) is added to the model to be trained, that is, the input is processed using the Dropout regularization strategy.

[0137] It should be noted that for the same input and the same model to be trained, two different output distributions are obtained through the two Dropout regularization strategies. Specifically, the neurons dropped each time are basically different, based on which the model to be trained obtains two different output distributions.

[0138] For ease of understanding, a sample set is given as:

[0139]

[0140] Thus, the two output distributions can be recorded as:

[0141] P θ (y|χ) and P′ θ (y|χ).

[0142] Step 233, calculating the KL divergence and cross entropy loss of the two output distributions, and calculating the total loss of the KL divergence and the cross entropy loss;

[0143] Among them, KL divergence can be used as the loss of model training, which is used to characterize the difference between the two output distributions.

[0144] In this embodiment, two cross entropy losses are included, that is, one output distribution corresponds to one cross entropy loss. The total loss is obtained by adding the two cross entropy losses and the KL divergence.

[0145] Step 234: Based on the total loss, the model to be trained is trained to obtain a quality evaluation model.

[0146] Specifically, based on a preset loss function, the model to be trained is trained to obtain a quality evaluation model. The preset loss function includes a KL divergence function and a cross entropy loss function.

[0147] In this embodiment, the total loss is obtained based on the KL divergence and cross entropy loss of the two output distributions, and then the model is trained based on the total loss to continuously lower the KL divergence between the two output distributions during training, and indirectly use the overall symmetric KL divergence by exchanging the positions of the two output distributions, thereby reducing the randomness of Dropout. That is, by designing the loss function, the inconsistency of Dropout during training and testing is compensated, thereby constraining the randomness of Dropout to the network, reducing the contribution of easy-to-learn samples to the loss, and focusing on difficult-to-learn samples, thereby optimizing the training results, so as to train a quality evaluation model with higher performance, and ultimately further improve the accuracy of medical literature quality evaluation.

[0148] The medical literature evaluation device provided by the present invention is described below. The medical literature evaluation device described below and the medical literature evaluation method described above can be referenced to each other.

[0149] The medical literature evaluation device provided by the present invention comprises:

[0150] An acquisition module, used for acquiring the text to be evaluated of the medical literature to be evaluated;

[0151] An evaluation module is used to input the text to be evaluated into a quality evaluation model, perform quality evaluation, and obtain a quality evaluation result output by the quality evaluation model. The quality evaluation model is trained based on sentence vector similarity results. The sentence vector similarity results are obtained by calculating the similarity of sentence vectors generated by a sentence vector generation model of a twin network structure. The sentence vector generation model includes a biomedical pre-training model.

[0152] The medical literature evaluation device also includes a training module, which is used to obtain a first sentence text and a second sentence text as training data, input the first sentence text and the second sentence text into a sentence vector generation model of a twin network structure, perform feature extraction, and obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, wherein the sentence vector generation model includes a biomedical pre-training model; perform similarity calculation on the first sentence vector and the second sentence vector to obtain a sentence vector similarity result; and based on the sentence vector similarity result, train the model to be trained to obtain a quality evaluation model.

[0153] The sentence vector generation model includes a first sentence vector generation sub-model and a second sentence vector generation sub-model. The training module is also used to input the first sentence text into the first sentence vector generation sub-model, perform feature extraction, and obtain the first sentence vector output by the first sentence vector generation sub-model; input the second sentence text into the second sentence vector generation sub-model, perform feature extraction, and obtain the second sentence vector output by the second sentence vector generation sub-model.

[0154] The first sentence vector generation sub-model includes a biomedical pre-training model and a pooling layer. The training module is also used to input the first sentence text into the biomedical pre-training model, perform feature extraction, and obtain a first feature vector output by the biomedical pre-training model; input the first feature vector into the pooling layer, perform pooling processing, and obtain a first sentence vector output by the pooling layer.

[0155] The first feature vector is a first word vector, and the training module is further used to input the first word vector into the pooling layer, perform average pooling processing, and obtain the first sentence vector output by the pooling layer.

[0156] The training module is also used to obtain a distance vector between the first sentence vector and the second sentence vector; concatenate the first sentence vector, the second sentence vector and the distance vector to obtain an aggregate vector; and based on the sentence vector similarity result and the aggregate vector, train the model to be trained to obtain a quality evaluation model.

[0157] The model to be trained includes a discarding layer Dropout, and the training module is also used to obtain two output distributions of the same input based on the Dropout; calculate the KL divergence and cross entropy loss of the two output distributions, and calculate the total loss of the KL divergence and the cross entropy loss; based on the total loss, train the model to be trained to obtain a quality evaluation model.

[0158] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the medical literature evaluation method, which includes: obtaining the text to be evaluated of the medical literature to be evaluated; inputting the text to be evaluated into the quality evaluation model, performing quality evaluation, and obtaining the quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is trained based on the sentence vector similarity result, and the sentence vector similarity result is obtained by calculating the similarity of the sentence vector generated by the sentence vector generation model of the twin network structure, and the sentence vector generation model includes a biomedical pre-training model.

[0159] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0160] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the medical literature evaluation method provided by the above methods, and the method includes: obtaining the text to be evaluated of the medical literature to be evaluated; inputting the text to be evaluated into a quality evaluation model, performing quality evaluation, and obtaining the quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is trained based on the sentence vector similarity result, and the sentence vector similarity result is obtained by performing similarity calculation on the sentence vector generated by the sentence vector generation model of the twin network structure, and the sentence vector generation model includes a biomedical pre-training model.

[0161] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the medical literature evaluation method provided by the above-mentioned methods, the method comprising: obtaining a text to be evaluated of the medical literature to be evaluated; inputting the text to be evaluated into a quality evaluation model, performing quality evaluation, and obtaining a quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is trained based on a sentence vector similarity result, and the sentence vector similarity result is obtained by performing similarity calculation on sentence vectors generated by a sentence vector generation model of a twin network structure, and the sentence vector generation model comprises a biomedical pre-training model.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical literature evaluation method, characterized in that: include: Obtain the text to be reviewed of the medical literature to be reviewed; Inputting the text to be evaluated into a quality evaluation model to perform quality evaluation, and obtaining a quality evaluation result output by the quality evaluation model, wherein the quality evaluation model is trained based on sentence vector similarity results, and the sentence vector similarity results are obtained by calculating the similarity of sentence vectors generated by a sentence vector generation model of a twin network structure, and the sentence vector generation model includes a biomedical pre-training model; Also included is a training method for the quality assessment model: Acquire a first sentence text and a second sentence text as training data, input the first sentence text and the second sentence text into a sentence vector generation model of a twin network structure, perform feature extraction, and obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, wherein the sentence vector generation model includes a biomedical pre-training model; Calculate the similarity between the first sentence vector and the second sentence vector to obtain a sentence vector similarity result; Based on the sentence vector similarity results, the model to be trained is trained to obtain a quality evaluation model; The sentence vector generation model includes a first sentence vector generation sub-model and a second sentence vector generation sub-model, and the first sentence text and the second sentence text are input into the sentence vector generation model of the twin network structure, and feature extraction is performed to obtain a first sentence vector corresponding to the first sentence text and a second sentence vector corresponding to the second sentence text, including: Inputting the first sentence text into the first sentence vector generation sub-model, performing feature extraction, and obtaining a first sentence vector output by the first sentence vector generation sub-model; Inputting the second sentence text into the second sentence vector generation sub-model, performing feature extraction, and obtaining a second sentence vector output by the second sentence vector generation sub-model; The first sentence vector generation sub-model includes a biomedical pre-training model and a pooling layer, and the first sentence text is input into the first sentence vector generation sub-model to perform feature extraction to obtain a first sentence vector output by the first sentence vector generation sub-model, including: Inputting the first sentence text into the biomedical pre-training model, performing feature extraction, and obtaining a first feature vector output by the biomedical pre-training model; Inputting the first feature vector into the pooling layer, performing pooling processing, and obtaining a first sentence vector output by the pooling layer; The first feature vector is a first word vector, and the first feature vector is input into the pooling layer for pooling processing to obtain a first sentence vector output by the pooling layer, including: The first word vector is input into the pooling layer, and average pooling is performed to obtain the first sentence vector output by the pooling layer.

2. The medical literature evaluation method according to claim 1, characterized in that: Before training the model to be trained to obtain the quality evaluation model based on the sentence vector similarity result, the method further includes: Obtaining a distance vector between the first sentence vector and the second sentence vector; Concatenate the first sentence vector, the second sentence vector and the distance vector to obtain an aggregated vector; The step of training the model to be trained based on the sentence vector similarity result to obtain a quality evaluation model includes: Based on the sentence vector similarity result and the aggregated vector, the model to be trained is trained to obtain a quality evaluation model.

3. The medical literature evaluation method according to claim 1 or 2, characterized in that: The model to be trained includes a dropout layer, and the quality evaluation model is obtained by training the model to be trained, including: Based on the Dropout, two output distributions of the same input are obtained; Calculating the KL divergence and the cross entropy loss of the two output distributions, and calculating the total loss of the KL divergence and the cross entropy loss; Based on the total loss, the model to be trained is trained to obtain a quality evaluation model.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the medical literature evaluation method according to any one of claims 1 to 3 are implemented.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the medical literature evaluation method according to any one of claims 1 to 3 are implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the medical literature evaluation method according to any one of claims 1 to 3 are implemented.

Citation Information

Patent Citations

  • Key information extraction method and device, electronic equipment and storage medium

    CN112784601A

  • Text automatic scoring method and device based on twin neural network and electronic equipment

    CN112989784A

  • Literature knowledge context generation method and device and storage medium

    CN113076432A

  • Text classification method, computer device, and storage medium

    US20200293720A1