Word weight sorting method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a word weight sorting method and device, equipment and a storage medium, and the sorting method comprises the steps: obtaining a first target statement, and carrying out the word segmentation of the first target statement, and obtaining a first vocabulary set contained in the first target statement; removing each first vocabulary in the first vocabulary set from the first target statement to obtain a second target statement; forming any second target statement and the first target statement into a sentence pair, inputting the sentence pair into a pre-training language model, and generating a vector representation corresponding to the sentence pair; according to the vector representation corresponding to the sentence pair, determining a target similarity between a second target statement and the first target statement in the sentence pair; sorting the target similarities of the second target statements and the first target statements; and determining a weight sequence of each first vocabulary according to the sequence of each target similarity. The problem that in the prior art, word weights cannot be accurately sorted is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method, device, equipment and storage medium for sorting word weights. Background Art

[0002] Word weight is a common basic task in the fields of information retrieval and natural language processing. Its core goal is to determine the importance of each word in an input text, so as to provide assistance and guidance for subsequent related tasks. Especially in the financial-related fields, such as the insurance field, word weight is playing an increasingly important role.

[0003] Currently, the mainstream technologies of word weight can be divided into two categories: supervised and unsupervised. The unsupervised method does not require labeled data. The unsupervised method usually determines the importance of words based on some statistical characteristics or language structure rules of the text itself. For example, a common unsupervised strategy is to measure the weight of a word according to its frequency of occurrence in the text. Generally, it is considered that words with a higher frequency of occurrence may have relatively higher importance. However, this simple frequency statistics method has obvious drawbacks. On the one hand, function words such as "de", "shi", "zai" etc. often have extremely high frequencies of occurrence in the text, but they do not contribute much to the core semantic expression of the text. If they are simply given high weights based on frequency, it will interfere with the grasping of key information in the text. On the other hand, some professional terms or specific topic words that are of great significance in a specific context but have a relatively low overall frequency of occurrence may be ignored due to their low frequency. Therefore, due to the lack of more accurate understanding and differentiation of semantics, the effect that the unsupervised method can achieve is relatively limited. In those scenarios where high precision of word weight is required, such as professional literature retrieval, accurate text classification and analysis tasks, etc., the unsupervised method is difficult to directly and effectively complete the task by itself and cannot meet the high-precision requirements, and other technologies or methods need to be used to further optimize and improve the determination of word weight.

[0004] Although the supervised method can achieve high precision, it often requires a large amount of data for training. Since the word weight task requires marking the weight of each word in the text one by one, this makes the annotation work extremely time-consuming and laborious, and the cost is quite high. Especially in the open-domain scenario, a huge amount of training data is required, which is extremely difficult in the industrial field. Therefore, it is relatively more common to carry out word weight applications in vertical fields for specific fields. Even in the closed-domain scenario, it is not easy to obtain a large amount of labeled data, and the high-quality data open-sourced on the Internet is also quite scarce, which is difficult to give substantial and powerful support. Therefore, it is still difficult to achieve accurate sorting of word weights using the supervised method. Summary of the Invention

[0005] The present invention provides a word weight sorting method, device, equipment and storage medium to solve the problem that word weights cannot be accurately sorted in the prior art.

[0006] In a first aspect, a method for ranking word weights is provided, comprising:

[0007] Acquire a first target sentence, segment the first target sentence, and obtain a first vocabulary set included in the first target sentence;

[0008] Removing each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0009] Combining any of the second target sentences and the first target sentence into a sentence pair, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair;

[0010] Determining a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair;

[0011] sorting the target similarities between each of the second target sentences and the first target sentence;

[0012] The weight ranking of each of the first words is determined according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0013] In a second aspect, a word weight sorting device is provided, comprising:

[0014] An acquisition module, configured to acquire a first target sentence, and segment the first target sentence to obtain a first vocabulary included in the first target sentence;

[0015] a removal module, configured to remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentence is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0016] A generation module, configured to combine any second target sentence and the first target sentence into a sentence pair, and input the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair;

[0017] A first determination module, configured to determine a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair;

[0018] A sorting module, used to sort the target similarities between each of the second target sentences and the first target sentence;

[0019] The second determination module is used to determine the weight ranking of each of the first words according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0020] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned word weight sorting method when executing the computer program.

[0021] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned word weight sorting method are implemented.

[0022] In the scheme implemented by the above-mentioned word weight sorting method, when the sentence pair formed by two sentences is input into the pre-trained language model, the vector representation of the sentence pair can be obtained, and then the similarity of the two sentences can be obtained, and finally the similarity of each second target sentence with the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first vocabulary removed from the first target sentence by the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first vocabulary removed from the first target sentence by the second target sentence. In this way, the similarity of each second target sentence with the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first vocabulary in the first target sentence. In this way, since this scheme is a supervised learning model, it can avoid the problem of low precision of sorting word weights caused by unsupervised methods in the prior art. In addition, this scheme does not need a large amount of training data like the supervised method in the prior art, so that the defect of labeling weights of each word in the text one by one is required, so this scheme can avoid the problem of time-consuming and difficult to obtain data labeling in the supervised method in the prior art. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0024] Figure 1 is a flow chart of a method for ranking word weights in one embodiment of the present invention;

[0025] Figure 2 is another flow chart of a method for ranking word weights in one embodiment of the present invention;

[0026] Figure 3 It is another flow chart of a method for sorting word weights in one embodiment of the present invention;

[0027] Figure 4 is a structural schematic diagram of a device for sorting word weights in one embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0029] Figure 6 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 As shown, Figure 1 A flow chart of a method for sorting word weights provided in an embodiment of the present invention includes the following steps:

[0032] Step S10: acquiring a first target sentence, segmenting the first target sentence, and obtaining a first vocabulary set included in the first target sentence.

[0033] It should be noted that, in step S10, the first target sentence can be any sentence that needs to be sorted by word weights. In step S10, the way of segmenting the first target sentence can be any feasible way, for example, it can be a dictionary matching method, it can also be a minimum segmentation method, it can also be a maximum entropy model segmentation method, etc.

[0034] It can be understood that the first vocabulary set finally obtained by segmenting the first target sentence is also the set of all the vocabulary in the first target sentence. For example, the first target sentence is "the vehicle insurance amount is 50,000 yuan", and all the words obtained by segmentation are "vehicle", "insurance", "amount", "for" and "50,000 yuan", so the first vocabulary set is "vehicle", "insurance", "amount", "for" and "50,000 yuan".

[0035] Step S20: remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words.

[0036] It should be noted that, in step S20, when a first word in the first vocabulary set is removed from the first target sentence, a second target sentence (a sentence after the first word is removed from the first target sentence) is formed. In this way, each time a first word in the first vocabulary set is removed from the first target sentence, a second target sentence equal to the number of first words is formed.

[0037] As a specific example, the first target sentence is "the insurance amount of the vehicle is 50,000 yuan", and the first vocabulary set obtained by word segmentation is "vehicle", "insurance", "amount", "for" and "50,000 yuan". When the first word "vehicle" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "the insurance amount is 50,000 yuan", when the first word "insurance" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "the amount of the vehicle is 50,000 yuan", when the first word "amount" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "the insurance of the vehicle is 50,000 yuan", when the first word "for" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "the insurance amount of the vehicle is 50,000 yuan", when the first word "50,000 yuan" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "the insurance amount of the vehicle is", and finally the second target sentences are combined.

[0038] In one embodiment of the present invention, Figure 2 A flowchart of another method for sorting word weights provided by an embodiment of the present invention, such as Figure 2 As shown, after step S20 and before step S40, the following further comprises:

[0039] Step S31, obtaining a first statement;

[0040] Step S32, generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold;

[0041] Step S33, forming a first triplet of data from the first sentence, the second sentence and the third sentence, and inputting the first triplet of data into a language model to be trained to train the language model to be trained;

[0042] Step S34, calculating the output value of the loss function of the language model to be trained during the training process, until the output value of the loss function reaches a preset value, and determining the language model to be trained as a pre-trained language model.

[0043] It should be noted that the first sentence obtained in step S31 can be any sentence, and the number of first sentences can be any number, which is not limited in detail here.

[0044] It should be noted that the method of generating a second sentence whose semantic similarity with the first sentence is higher than the first preset threshold can be any feasible method. For example, some words in the first sentence can be replaced with synonyms to obtain a new sentence, and then the similarity between the new sentence and the first sentence is evaluated. If the similarity is higher than the first preset threshold, the new sentence is determined as the second sentence. If it is not higher than the first preset threshold, other words in the first sentence are selected to replace with synonyms to generate a new sentence, and the similarity between the new sentence and the first sentence is evaluated. This process is repeated until the similarity between the new sentence and the first sentence is higher than the first preset threshold to obtain the second sentence. Similarly, it can be understood that when obtaining the third sentence, some words in the first sentence can also be replaced with synonyms to obtain the third sentence.

[0045] In this way, in step S33, the first statement, the second statement and the third statement can be formed into a first triple data. For example, the first triple data can be shown as (first statement, second statement, third statement).

[0046] It can be understood that in step S34, after the first triple data is input into the language model to be trained, the language model to be trained will output the similarity between the first sentence and the second sentence, as well as the similarity between the first sentence and the third sentence. In this way, by comparing the similarity between the first sentence and the second sentence output by the language model to be trained, and the actual similarity between the first sentence and the second sentence, as well as comparing the similarity between the first sentence and the third sentence output by the language model to be trained, and the actual similarity between the first sentence and the third sentence, the training effect of the language model to be trained can be obtained.

[0047] Furthermore, it is obvious that the training effect of the language model to be trained can be reflected by the output value of the loss function of the language model to be trained. When the output value of the loss function reaches the preset value, the language model to be trained can be determined as a pre-trained language model. It should be noted that the specific value of the preset value can be set according to the specific needs of the application.

[0048] It should be noted that in steps S31 to 34, the network structure of the pre-trained model can obtain the vector representation of the sentence pair, and after obtaining the vector representation of the sentence pair, the pre-trained model can also obtain the similarity between the two sentences in the sentence pair based on the vector representation of the sentence pair.

[0049] It can be understood that in step S31-step 34, a sentence similarity model can be trained, so that when two sentences are input into the sentence similarity model, the similarity of the two sentences can be obtained. Based on the sentence similarity model, the similarity between each second target sentence and the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first vocabulary removed from the first target sentence by the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first vocabulary removed from the first target sentence by the second target sentence. In this way, the similarity between each second target sentence and the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first vocabulary in the first target sentence. In this way, step S31-step S34 avoids the defect that the supervised method in the prior art requires a large amount of training data, thereby requiring the weight of each word in the text to be marked one by one, so that accurate sorting of word weights can be achieved.

[0050] In one embodiment of the present invention, the loss function is:

[0051] L=max(s(query,pos)-s(query,neg)+margin,0);

[0052] Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

[0053] It can be understood that the first sentence is the benchmark sentence and is the core object of the entire comparison process. The second sentence represents a sentence that is semantically similar to the first sentence, that is, a positive example. The third sentence is semantically different from the first sentence and is a negative example.

[0054] The specific calculation methods for the similarity between the first sentence and the second sentence, and the similarity between the first sentence and the third sentence can be implemented in many forms, such as cosine similarity calculation based on word vectors (converting words in the sentence into vector representations, and then calculating the cosine angle between the vectors to measure the similarity), semantic similarity score based on the output of a deep learning model (for example, using a pre-trained language model to extract the semantic representation of a sentence, and then determining the similarity through some distance metric), etc.

[0055] The preset hyperparameter is a constant parameter set in advance by humans. Its function is to increase the strictness of distinguishing positive examples from negative examples. From the expression of the loss function, it can be seen that even if s(query, pos) is larger than s(query, neg), if the difference does not exceed the value of margin, the loss value is still not 0, which prompts the model to learn a sufficiently large similarity gap between positive examples and negative examples to better distinguish them. For example, set margin to 0.5, if s(query, pos) is 0.6, s(query, neg) is 0.2, then s(query, pos)-s(query, neg) is equal to 0.4, which is less than margin, then the loss value will be max(0.4+0.5,0), that is, 0.9, and the model needs to continue to adjust the parameters to further increase this difference and reduce the loss value.

[0056] In short, by setting such a loss function, when training related models, we can use the similarity comparison between sentences, combined with the preset hyperparameters, to guide the model to learn and optimize in the direction of accurately distinguishing similar sentences (positive examples) and dissimilar sentences (negative examples), thereby improving the performance of the model on the corresponding tasks.

[0057] In one embodiment of the present invention, Figure 3 A flowchart of another method for sorting word weights provided for the implementation of the present invention is shown in FIG. Figure 3 As shown, after step S34, it also includes:

[0058] Step S35, marking the word weight data of the fourth sentence;

[0059] Step S36, extracting words whose word weight data is greater than a third preset threshold in the fourth sentence to obtain a fifth sentence;

[0060] Step S37, extracting words whose word weight data is less than a fourth preset threshold in the fourth sentence to obtain a sixth sentence;

[0061] Step S38, forming a second triplet of data from the fourth sentence, the fifth sentence and the sixth sentence, and inputting the second triplet of data as fine-tuning data into the pre-trained language model, fine-tuning the pre-trained language model, and obtaining a fine-tuned pre-trained language model.

[0062] It is understandable that in step S35, a small number of fourth sentences can be obtained, a small amount of word weight data can be annotated, and the relative order of the importance of the words in each fourth sentence can also be annotated. For each fourth sentence, each word is removed respectively to form several sentences that lack one of the words. Since the more important the removed word is, the lower the similarity between the sentence after the word is removed and the original sentence, multiple triples of data can be constructed. In this way, only a small number of fourth sentences need to be annotated to obtain fine-tuning data samples of the pre-trained language model. Fine-tuning the pre-trained language model using these fine-tuning data samples can make the model more accurate.

[0063] Step S40: any of the second target sentences and the first target sentence are combined into a sentence pair, and the sentence pair is input into a pre-trained language model to generate a vector representation corresponding to the sentence pair.

[0064] It should be noted that in step S40, the vector representation corresponding to the sentence pair is a vector representation that fully contains the feature information of the second target sentence and the first target sentence. The pre-trained language model is a model that can generate a vector representation corresponding to the sentence pair when the sentence pair is input into the pre-trained language model, which is not limited in detail here.

[0065] In one embodiment of the present invention, step S40 includes:

[0066] Step S41, input the sentence pair into the language model structure, process the sentence pair through the multi-head attention mechanism in the language model structure, and generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0067] Specifically, in step S41, the focus is on inputting the sentence pair into the language model structure and processing it with the help of the multi-head attention mechanism therein. The multi-head attention mechanism is an advanced and effective feature extraction and semantic association capture mechanism. When the sentence pair enters the language model structure, the multi-head attention mechanism begins to work. It will examine each word in the sentence pair (or more accurately, the word vector representation corresponding to the word) from multiple "heads" (that is, multiple different attention angles) at the same time, and calculate the degree of association between them. For example, for a word in a sentence pair, different "heads" may focus on words related to it in grammatical structure, words similar in semantics, words that are logically associated with it in the sentence pair, and so on. Through the above-mentioned processing process of the multi-head attention mechanism, it will adjust, weight, and other operations on the word vector of each position in the sentence pair, so that each word vector is integrated with the relevant semantic information from other parts of the sentence pair. Finally, all the processed word vectors in the entire sentence pair are arranged in order to form a feature representation sequence that integrates the semantic information of the sentence pair. This sequence is no longer a simple sequence of raw word vectors, but is the result of full consideration of the semantic interaction within the sentence pair. The vector at each position contains richer and more comprehensive semantic content, including both its own original semantics and the associated semantics with other words in the context of the entire sentence pair.

[0068] In step S41, the multi-head attention mechanism focuses on the semantic relationship in the sentence pair from multiple different angles, which can capture the complex semantic connection between sentences more comprehensively than the traditional single attention method. For example, a sentence pair may have both grammatical dependencies and deep semantic logical associations. Different "heads" can focus on these different aspects to ensure that important semantic information is not missed. In natural language, sometimes there is a long-distance semantic dependency between words. For example, in a long sentence or a complex sentence pair, the subject and the object may be far apart, but they have important semantic connections. The multi-head attention mechanism can cross these distances, accurately find and utilize this long-distance semantic dependency, so that the generated feature representation sequence can fully reflect the semantic picture of the sentence pair. The feature representation sequence generated by the multi-head attention mechanism, the vector of each position integrates multi-dimensional semantic information. This enables various subsequent tasks (such as classification, similarity calculation, etc.) based on the feature representation to utilize richer and more accurate semantic features, avoiding the one-sidedness that may be caused by using only single-dimensional semantic information, thereby improving the accuracy and effect of the task. Because the semantic information from each part of the sentence pair is integrated, the feature representation is more resistant to noise, local semantic ambiguity, etc. Even if the semantic expression of a part of the sentence pair is slightly vague or there are some interference factors, through the overall semantic fusion, relatively accurate and stable semantic features can still be extracted, which is conducive to the stable implementation of subsequent tasks. The feature representation sequence generated based on the multi-head attention mechanism can be easily applied to a variety of different natural language processing tasks because of its high quality and comprehensive capture of the semantic information of the sentence pair. Whether it is judging the category of sentence pairs in text classification tasks, measuring the similarity between sentence pairs in semantic similarity tasks, or mining the entity relationships contained in sentence pairs in relationship extraction tasks, this feature representation can provide strong semantic support without the need for large-scale feature adjustments or reconstructions for different tasks.

[0069] In one embodiment of the present invention, step S41 includes:

[0070] Step S411, input the sentence pair into a Transformer-based Cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0071] It is understandable that the Transformer architecture performs well in the field of natural language processing. The Transformer-based Cross-encoder can handle sequence data well, that is, text sequences such as natural language sentence pairs. It can effectively capture the semantic relationship within and between sentences through the self-attention mechanism. Compared with the traditional neural network architecture, the Transformer can better handle long sequence data, avoid the gradient vanishing or gradient explosion problems in the process of long sequence information transmission, and can perform parallel calculations to improve computational efficiency. The Cross-encoder is specifically used to encode sentence pairs. Its architectural design considers how to better integrate the semantic information between the two sentences. By inputting sentence pairs into it, its internal structure and parameters can be fully utilized to explore the grammatical, semantic, logical and other aspects of the relationship between the two sentences in the sentence pair, and generate a high-quality feature representation sequence that integrates semantic information. In the Cross-encoder, the two sentences in the sentence pair are not simply spliced ​​or processed separately, but have deep semantic interactions within the model. It can automatically learn the semantic correspondence between sentences. For example, for a question-answer sentence pair, it can accurately capture the semantic matching points between the question and the answer; for a sentence pair for semantic similarity judgment, it can find the key semantic components that reflect the similarity. This deep semantic interaction can generate a feature representation sequence that better reflects the overall semantics of the sentence pair, which is more conducive to subsequent natural language processing tasks than some simple splicing or shallow fusion methods.

[0072] Cross-encoder can accurately extract semantic features from sentence pairs. It can focus on representative semantic elements in sentence pairs, such as the relationship between entities, actions, attributes, etc., by performing multi-level encoding and feature extraction on sentence pairs. The feature representation sequence generated in this way not only contains the overall semantic information of the sentence pair, but also highlights the features of the key semantic parts, providing more valuable information for subsequent tasks (such as text classification, semantic matching, etc.).

[0073] The Transformer-based Cross-encoder has demonstrated good generalization capabilities in many natural language processing tasks. Whether it is text classification, semantic similarity calculation, information retrieval or relationship extraction, the generated feature representation sequence that integrates semantic information can be used as an effective input feature for further processing. This versatility means that it does not require large-scale architectural adjustments in different downstream tasks, but only requires appropriate fine-tuning according to the task objectives.

[0074] When faced with different task requirements and data characteristics, Cross-encoder can adapt relatively flexibly. For example, when the task focus changes from semantic similarity judgment to relationship extraction, it can use the generated feature representation sequence to meet the new task requirements by adjusting the model parameters or combining different loss functions, because it has already conducted a comprehensive and in-depth mining of the semantic information of the sentence pair.

[0075] Step S50: determining a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair.

[0076] It should be noted that, in step S50, the method of determining the target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector features corresponding to the sentence pair can be any feasible method. For example, the cosine similarity between the second target sentence and the first target sentence can be calculated based on the vector representation corresponding to the sentence pair.

[0077] In one embodiment of the present invention, step S50 includes:

[0078] Step S51, extracting the [CLS] representation in the vector representation;

[0079] Step S52: input the [CLS] representation into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

[0080] It can be understood that the [CLS] representation is a comprehensive representation of the overall semantics of the sentence pair. Extracting it can obtain the most critical semantic information of the sentence pair and avoid the local semantic interference that may exist in other position vectors. The fully connected layer can further perform nonlinear transformation on the extracted [CLS] representation, thereby mapping it to a suitable space, obtaining a matching score that is more in line with the semantic matching degree assessment, and enhancing the accuracy of similarity judgment.

[0081] Step S60: sorting the target similarities between each of the second target sentences and the first target sentences.

[0082] Step S70: determining the weight ranking of each of the first words according to the ranking of each of the target similarities, wherein the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0083] In this way, when the sentence pair formed by two sentences is input into the pre-trained language model, the vector representation of the sentence pair can be obtained, and then the similarity of the two sentences can be obtained, and finally the similarity of each second target sentence with the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first vocabulary removed from the first target sentence by the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first vocabulary removed from the first target sentence by the second target sentence. In this way, the similarity of each second target sentence with the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first vocabulary in the first target sentence. In this way, since this scheme is a supervised learning model, it can avoid the problem of low accuracy of sorting word weights caused by unsupervised methods in the prior art. In addition, this scheme does not need a large amount of training data like the supervised method in the prior art, so that the defect of labeling weights of each word in the text one by one is required, so this scheme can avoid the problem of time-consuming and difficult to obtain data labeling in the supervised method in the prior art. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted.

[0084] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0085] In one embodiment, a word weight sorting device is provided, and the word weight sorting device corresponds to the word weight sorting method in the above embodiment. Figure 4 As shown, the word weight ranking device includes an acquisition module 410, a removal module 420, a generation module 430, a first determination module 440, a ranking module 450, and a second determination module 460. The functional modules are described in detail as follows:

[0086] The acquisition module 410 is used to acquire a first target sentence, and segment the first target sentence to obtain a first vocabulary included in the first target sentence;

[0087] a removal module 420, configured to remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentence is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0088] A generating module 430, configured to combine any second target sentence and the first target sentence into a sentence pair, and input the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair;

[0089] A first determination module 440 is used to determine a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair;

[0090] A sorting module 450, configured to sort the target similarities between each of the second target sentences and the first target sentence;

[0091] The second determination module 460 is used to determine the weight ranking of each of the first words according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0092] In one embodiment, the removal module 420 is further configured to:

[0093] Get the first statement;

[0094] generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold;

[0095] The first sentence, the second sentence and the third sentence form a first triplet of data, and the first triplet of data is input into a language model to be trained, so as to train the language model to be trained;

[0096] The output value of the loss function of the language model to be trained during the training process is calculated until the output value of the loss function reaches a preset value, and the language model to be trained is determined as a pre-trained language model.

[0097] In one embodiment, the removal module 420 is further configured to:

[0098] Label the word weight data of the fourth sentence;

[0099] Extracting words in the fourth sentence whose word weight data is greater than a third preset threshold value to obtain a fifth sentence;

[0100] Extracting words whose word weight data is less than a fourth preset threshold value from the fourth sentence to obtain a sixth sentence;

[0101] The fourth sentence, the fifth sentence and the sixth sentence form a second triplet of data, and the second triplet of data is input into the pre-trained language model as fine-tuning data, and the pre-trained language model is fine-tuned to obtain a fine-tuned pre-trained language model.

[0102] In one embodiment, the generating module 430 is specifically configured to:

[0103] The sentence pair is input into a language model structure, and the sentence pair is processed through a multi-head attention mechanism in the language model structure to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0104] In one embodiment, the generating module 430 is further configured to:

[0105] The sentence pair is input into a Transformer-based Cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0106] In one embodiment, the first determining module 440 is specifically configured to:

[0107] Extracting a [CLS] representation from the vector representation;

[0108] The [CLS] representation is input into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

[0109] In one embodiment, the loss function is:

[0110] L=max(s(query,pos)-s(query,neg)+margin,0);

[0111] Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

[0112] The present invention provides a word weight sorting device. When a sentence pair formed by two sentences is input into a pre-trained language model, a vector representation of the sentence pair can be obtained, and then the similarity of the two sentences can be obtained, and finally the similarity of each second target sentence to the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first vocabulary removed from the first target sentence by the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first vocabulary removed from the first target sentence by the second target sentence. In this way, the similarity of each second target sentence to the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first vocabulary in the first target sentence. In this way, since this scheme is a supervised learning model, it can avoid the problem of low precision of sorting word weights caused by unsupervised methods in the prior art. In addition, this scheme does not need a large amount of training data like the supervised method in the prior art, so that the defect of labeling weights of each word in the text one by one is required, so this scheme can avoid the problem of time-consuming and difficult to obtain labeled data in the supervised method in the prior art. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted.

[0113] For the specific definition of the word weight sorting device, please refer to the definition of the word weight sorting method above, which will not be repeated here. The various modules in the above-mentioned word weight sorting device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] The ranking method based on the above word weights, such as Figure 5 As shown, an embodiment of the present invention further provides a structural schematic diagram of a word weight sorting device, which includes a processor 51 and a memory 52 coupled to the processor 51. The memory 52 stores a computer program, and when the computer program is executed by the processor 51, the processor 51 executes the steps of the word weight sorting method in the above embodiment.

[0115] For other details about how the processor 51 in the above-mentioned word weight sorting device implements the above-mentioned technical solution, please refer to the description of the word weight sorting method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0116] Among them, the processor 51 can also be called a CPU (Central Processing Unit), and the processor 51 may be an integrated circuit chip with signal processing capabilities; the processor 51 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 51 can also be any conventional processor, etc.

[0117] like Figure 6 As shown, the embodiment of the present invention also provides a schematic diagram of the structure of a computer-readable storage medium, on which a readable computer program 61 is stored; wherein the computer program 61 can be stored in the above storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0118] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional module in each embodiment of the present invention may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0121] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0122] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium), or a semiconductor medium (e.g., an SSD (solid state disk, solid state hard disk)).

[0123] The technical solution provided by the present invention is introduced in detail above. The present invention applies specific examples to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations. The non-Company software tools or components that appear in the embodiments of the present invention are merely examples and do not represent actual use.

Claims

1. A method for ranking word weights, characterized in that: include: Acquire a first target sentence, segment the first target sentence, and obtain a first vocabulary set included in the first target sentence; Removing each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words; Combining any of the second target sentences and the first target sentence into a sentence pair, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair; Determining a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair; sorting the target similarities between each of the second target sentences and the first target sentence; The weight ranking of each of the first words is determined according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

2. The method for ranking word weights according to claim 1, characterized in that: After removing a random first word in the first vocabulary set from the first target sentence to obtain a second target sentence until the number of the obtained second target sentences is equal to the number of the first vocabulary, before forming a sentence pair with any second target sentence and the first target sentence, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair, the method further includes: Get the first statement; generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold; The first sentence, the second sentence and the third sentence form a first triplet of data, and the first triplet of data is input into a language model to be trained, so as to train the language model to be trained; The output value of the loss function of the language model to be trained during the training process is calculated until the output value of the loss function reaches a preset value, and the language model to be trained is determined as a pre-trained language model.

3. The method for ranking word weights according to claim 2, characterized in that: After the output value of the loss function reaches a preset value and the language model to be trained is determined as a pre-trained language model, the method further includes: Label the word weight data of the fourth sentence; Extracting words in the fourth sentence whose word weight data is greater than a third preset threshold value to obtain a fifth sentence; Extracting words whose word weight data is less than a fourth preset threshold value from the fourth sentence to obtain a sixth sentence; The fourth sentence, the fifth sentence and the sixth sentence form a second triplet of data, and the second triplet of data is input into the pre-trained language model as fine-tuning data, and the pre-trained language model is fine-tuned to obtain a fine-tuned pre-trained language model.

4. The method for ranking word weights according to claim 1, characterized in that: The step of forming a sentence pair from any of the second target sentences and the first target sentence, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair includes: The sentence pair is input into a language model structure, and the sentence pair is processed through a multi-head attention mechanism in the language model structure to generate a feature representation sequence that integrates the semantic information of the sentence pair.

5. The method for ranking word weights according to claim 4, characterized in that: The step of inputting the sentence pair into a language model structure, processing the sentence pair through a multi-head attention mechanism in the language model structure, and generating a feature representation sequence integrating semantic information of the sentence pair includes: The sentence pair is input into a Transformer-based Cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

6. The method for ranking word weights according to claim 5, characterized in that: The determining, according to the vector representation corresponding to the sentence pair, a target similarity between the second target sentence and the first target sentence in the sentence pair, comprises: Extracting a [CLS] representation from the vector representation; The [CLS] representation is input into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

7. The method for ranking word weights according to claim 2, characterized in that: The loss function is: L=max(s(query,pos)-s(query,neg)+margin,0); Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

8. A word weight sorting device, characterized in that: include: An acquisition module, configured to acquire a first target sentence, and segment the first target sentence to obtain a first vocabulary included in the first target sentence; a removal module, configured to remove each first word in the first target sentence respectively, to obtain a second target sentence corresponding to each first word; A generation module, configured to combine any second target sentence and the first target sentence into a sentence pair, and input the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair; A first determination module, configured to determine a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair; A sorting module, used to sort the target similarities between each of the second target sentences and the first target sentence; The second determination module is used to determine the weight ranking of each of the first words according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for sorting word weights as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for ranking word weights as claimed in any one of claims 1 to 7 are implemented.

Images