Word weight ranking method, device and equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a word weight sorting method, device and equipment and a storage medium, the sorting method comprising the following steps: obtaining a first target sentence, performing word segmentation on the first target sentence to obtain a first vocabulary set contained in the first target sentence; removing each first vocabulary in the first vocabulary set from the first target sentence to obtain a second target sentence; forming a sentence pair by combining any second target sentence and the first target sentence, inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair; determining a target similarity between the second target sentence and the first target sentence in the sentence pair according to the vector representation corresponding to the sentence pair; sorting the target similarities of the second target sentences and the first target sentence; and determining a weight order of each first vocabulary according to the sorting of the target similarities. The application solves the problem that the word weight cannot be accurately sorted in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method, device, equipment and storage medium for sorting word weights. Background Art

[0002] Word weight is a common basic task in the fields of information retrieval and natural language processing. Its core goal is to determine the importance of each word in an input text, so as to provide assistance and guidance for subsequent related tasks. Especially in financial-related fields, such as the insurance field, word weight is playing an increasingly important role.

[0003] Currently, the mainstream technologies of word weight can be divided into two categories: supervised and unsupervised. Unsupervised methods do not require labeled data. Unsupervised methods usually determine the importance of words based on some statistical characteristics or language structure rules of the text itself. For example, a common unsupervised strategy is to measure the weight of a word according to its frequency of occurrence in the text. Generally, it is considered that words with higher frequencies of occurrence may have relatively higher importance. However, this simple frequency statistics method has obvious drawbacks. On the one hand, function words such as "de", "shi", "zai", etc. often appear very frequently in the text, but they do not contribute much to the core semantic expression of the text. If they are simply given high weights based on frequency, it will interfere with the grasp of key information in the text. On the other hand, some professional terms or specific topic words that are of great significance in a specific context but have a relatively low overall frequency of occurrence may be ignored due to their low frequency. Therefore, due to the lack of more accurate understanding and differentiation of semantics, the effects that unsupervised methods can achieve are relatively limited. In scenarios where high precision of word weight is required, such as professional literature retrieval, accurate text classification and analysis tasks, etc., unsupervised methods are difficult to directly and effectively complete tasks by themselves and cannot meet the high-precision requirements, and other technologies or methods need to be used to further optimize and improve the determination of word weight.

[0004] Although supervised methods can achieve high precision, they often require a large amount of data for training. Since the word weight task requires marking the weight of each word in the text one by one, this makes the annotation work extremely time-consuming, laborious and costly. Especially in open-domain scenarios, a huge amount of training data is required, which is extremely difficult in the industrial community. Therefore, it is relatively more common to carry out word weight applications in vertical fields for specific fields. Even in closed-domain scenarios, it is not easy to obtain a large amount of labeled data, and high-quality open-source data on the network is also quite scarce, which is difficult to provide substantial and powerful support. Therefore, it is still difficult to achieve accurate sorting of word weights using supervised methods. Summary of the Invention

[0005] The present invention provides a word weight sorting method, device, equipment and storage medium to solve the problem in the prior art that word weights cannot be accurately sorted.

[0006] In a first aspect, a method for ranking word weights is provided, comprising:

[0007] Acquire a first target sentence, segment the first target sentence into words, and obtain a first vocabulary set included in the first target sentence;

[0008] removing each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0009] Combining any second target sentence with the first target sentence into a sentence pair, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair;

[0010] determining a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair;

[0011] sorting the target similarities between each of the second target sentences and the first target sentences;

[0012] The weight ranking of each of the first words is determined according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0013] In a second aspect, a device for sorting word weights is provided, comprising:

[0014] an acquisition module, configured to acquire a first target sentence, and segment the first target sentence to obtain a first vocabulary contained in the first target sentence;

[0015] a removal module, configured to remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0016] a generation module, configured to combine any second target sentence and the first target sentence into a sentence pair, input the sentence pair into a pre-trained language model, and generate a vector representation corresponding to the sentence pair;

[0017] a first determining module, configured to determine a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair;

[0018] a sorting module, configured to sort the target similarities between each of the second target sentences and the first target sentence;

[0019] The second determining module is configured to determine a weight ranking of each of the first words according to the ranking of each of the target similarities, wherein the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0020] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned word weight sorting method when executing the computer program.

[0021] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned word weight sorting method are implemented.

[0022] In the scheme implemented by the above-mentioned word weight ranking method, when a sentence pair formed by two sentences is input into the pre-trained language model, a vector representation of the sentence pair can be obtained, and then the similarity of the two sentences can be obtained, and finally the similarity of each second target sentence to the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first word removed from the second target sentence from the first target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first word removed from the second target sentence from the first target sentence. In this way, the similarity of each second target sentence to the first target sentence is ranked, and the reverse order of the ranking obtained is the importance ranking of each first word in the first target sentence. In this way, since this scheme is a supervised learning model, it can avoid the problem of low word weight ranking accuracy caused by unsupervised methods in the prior art. In addition, this scheme does not require a large amount of training data like supervised methods in the prior art, which requires the defect of labeling weights for each word in the text one by one. Therefore, this scheme can avoid the problem of supervised methods in the prior art that labeling data is time-consuming and difficult to obtain. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 1 is a flow chart of a method for ranking word weights according to an embodiment of the present invention;

[0025] Figure 2 is another flow chart of a method for ranking word weights according to an embodiment of the present invention;

[0026] Figure 3 This is another flow chart of a method for ranking word weights in one embodiment of the present invention;

[0027] Figure 4 1 is a schematic structural diagram of a device for sorting word weights according to an embodiment of the present invention;

[0028] Figure 5 is a structural diagram of a computer device in one embodiment of the present invention;

[0029] Figure 6 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] See also Figure 1 As shown, Figure 1 A flowchart of a method for ranking word weights provided in an embodiment of the present invention includes the following steps:

[0032] Step S10: Acquire a first target sentence, segment the first target sentence, and obtain a first vocabulary set included in the first target sentence.

[0033] It should be noted that in step S10, the first target sentence can be any sentence that needs to be sorted by word weights. In step S10, the word segmentation method for the first target sentence can be any feasible method, for example, a dictionary matching method, a minimum segmentation method, or a maximum entropy model word segmentation method.

[0034] It is understandable that the first vocabulary set ultimately obtained by segmenting the first target sentence is also the set of all vocabulary in the first target sentence. For example, if the first target sentence is "The vehicle insurance amount is 50,000 yuan," all the vocabulary obtained by segmentation are "vehicle," "insurance," "amount," "for," and "50,000 yuan," so the first vocabulary set is "vehicle," "insurance," "amount," "for," and "50,000 yuan."

[0035] Step S20: Remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence. The number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words.

[0036] It should be noted that in step S20, when a first word in the first vocabulary set is removed from the first target sentence, a second target sentence (the sentence after the first word is removed from the first target sentence) is formed. In this way, each time a first word in the first vocabulary set is removed from the first target sentence, a second target sentence equal in number to the first word is formed.

[0037] As a specific example, the first target sentence is "The insurance amount of the vehicle is 50,000 yuan", and the first vocabulary set obtained by word segmentation is "vehicle", "insurance", "amount", "for", and "50,000 yuan". When the first word "vehicle" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "The insurance amount is 50,000 yuan", when the first word "insurance" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "The vehicle amount is 50,000 yuan", when the first word "amount" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "The vehicle insurance is 50,000 yuan", when the first word "for" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "The vehicle insurance amount is 50,000 yuan", when the first word "50,000 yuan" in the first vocabulary set is removed from the first target sentence, the second target sentence formed is "The vehicle insurance amount is", and finally the second target sentences are combined.

[0038] In one embodiment of the present invention, Figure 2 A flowchart of another word weight sorting method provided by an embodiment of the present invention, such as Figure 2 As shown, after step S20 and before step S40, the following further comprises:

[0039] Step S31, obtaining a first statement;

[0040] Step S32, generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold;

[0041] Step S33, forming a first triplet of data from the first sentence, the second sentence, and the third sentence, and inputting the first triplet of data into a language model to be trained to train the language model to be trained;

[0042] Step S34: Calculate the output value of the loss function of the language model to be trained during the training process until the output value of the loss function reaches a preset value, and determine the language model to be trained as a pre-trained language model.

[0043] It should be noted that the first sentence obtained in step S31 can be any sentence, and the number of first sentences can be any number, which is not limited in detail here.

[0044] It should be noted that the method of generating a second sentence having a semantic similarity with the first sentence higher than the first preset threshold can be any feasible method. For example, some words in the first sentence can be replaced with synonyms to obtain a new sentence, and then the similarity between the new sentence and the first sentence is evaluated. If the similarity is higher than the first preset threshold, the new sentence is determined to be the second sentence. If it is not higher than the first preset threshold, other words in the first sentence are selected to replace synonyms to generate a new sentence, and the similarity between the new sentence and the first sentence is evaluated. This process is repeated until the similarity between the new sentence and the first sentence is higher than the first preset threshold to obtain the second sentence. Similarly, it can be understood that when obtaining the third sentence, some words in the first sentence can also be replaced with synonyms to obtain the third sentence.

[0045] In this way, in step S33, the first statement, the second statement and the third statement can be formed into a first triple data. For example, the first triple data can be shown as (first statement, second statement, third statement).

[0046] It can be understood that in step S34, after the first triple data is input into the language model to be trained, the language model to be trained will output the similarity between the first sentence and the second sentence, as well as the similarity between the first sentence and the third sentence. In this way, by comparing the similarity between the first sentence and the second sentence output by the language model to be trained, and the actual similarity between the first sentence and the second sentence, as well as comparing the similarity between the first sentence and the third sentence output by the language model to be trained, and the actual similarity between the first sentence and the third sentence, the training effect of the language model to be trained can be obtained.

[0047] Furthermore, it is clear that the training effect of the language model to be trained can be reflected by the output value of the loss function of the language model to be trained. When the output value of the loss function reaches a preset value, the language model to be trained can be determined as a pre-trained language model. It should be noted that the specific value of the preset value can be set according to the specific needs of the application.

[0048] It should be noted that in steps S31 to 34, the network structure of the pre-trained model can obtain the vector representation of the sentence pair, and after obtaining the vector representation of the sentence pair, the pre-trained model can also obtain the similarity between the two sentences in the sentence pair based on the vector representation of the sentence pair.

[0049] It is understandable that in steps S31-34, a sentence similarity model can be trained so that when two sentences are input into the sentence similarity model, the similarity of the two sentences can be obtained. Based on the sentence similarity model, the similarity between each second target sentence and the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first vocabulary removed from the first target sentence in the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first vocabulary removed from the first target sentence in the second target sentence. In this way, the similarity between each second target sentence and the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first vocabulary in the first target sentence. In this way, steps S31-S34 avoid the defect of the supervised method in the prior art that a large amount of training data is required, and thus the weight of each word in the text needs to be labeled one by one, so that accurate sorting of word weights can be achieved.

[0050] In one embodiment of the present invention, the loss function is:

[0051] L=max(s(query,pos)-s(query,neg)+margin,0);

[0052] Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

[0053] As you can understand, the first statement serves as the benchmark and is the core object of the entire comparison process. The second statement represents a statement that is semantically similar to the first statement and is therefore a positive example. The third statement is semantically quite different from the first statement and is therefore a negative example.

[0054] The specific calculation methods for the similarity between the first sentence and the second sentence, and the similarity between the first sentence and the third sentence can be implemented in various forms, such as cosine similarity calculation based on word vectors (converting words in the sentence into vector representations, and then calculating the cosine angle between the vectors to measure the similarity), semantic similarity score based on the output of a deep learning model (for example, using a pre-trained language model to extract the semantic representation of the sentence, and then determining the similarity through some distance metric), etc.

[0055] The preset hyperparameter is a constant parameter set in advance by humans. Its function is to increase the strictness of distinguishing between positive and negative examples. From the expression of the loss function, it can be seen that even if s(query,pos) is larger than s(query,neg), if the difference does not exceed the value of margin, the loss value is still not 0. This forces the model to learn a sufficiently large similarity gap between positive and negative examples to better distinguish them. For example, if the margin is set to 0.5, if s(query,pos) is 0.6 and s(query,neg) is 0.2, then s(query,pos)-s(query,neg) is equal to 0.4, which is less than the margin. Then the loss value will be max(0.4+0.5,0), that is, 0.9. The model needs to continue to adjust the parameters to further increase this difference and reduce the loss value.

[0056] In short, by setting such a loss function, when training related models, we can use the similarity comparison between sentences, combined with the preset hyperparameters, to guide the model to learn and optimize in the direction of accurately distinguishing similar sentences (positive examples) and dissimilar sentences (negative examples), thereby improving the performance of the model on the corresponding tasks.

[0057] In one embodiment of the present invention, Figure 3 A flowchart of another word weight sorting method provided for the implementation of the present invention is as follows: Figure 3 As shown, after step S34, the following steps are further included:

[0058] Step S35, marking the word weight data of the fourth sentence;

[0059] Step S36, extracting words in the fourth sentence whose word weight data is greater than a third preset threshold to obtain a fifth sentence;

[0060] Step S37, extracting words in the fourth sentence whose word weight data is less than a fourth preset threshold to obtain a sixth sentence;

[0061] Step S38: Form the fourth sentence, the fifth sentence, and the sixth sentence into a second triplet of data, and input the second triplet of data as fine-tuning data into the pre-trained language model, fine-tune the pre-trained language model, and obtain a fine-tuned pre-trained language model.

[0062] It is understood that in step S35, a small number of fourth sentences can be obtained and labeled with a small amount of word weight data. The relative order of the importance of the words in each fourth sentence can also be labeled. For each fourth sentence, each word is removed to construct several sentences that lack one of the words. Since the more important the removed word, the lower the similarity between the sentence after removing the word and the original sentence, multiple triples of data can be constructed. In this way, only a small number of fourth sentences need to be labeled to obtain fine-tuning data samples for the pre-trained language model. Using these fine-tuning data samples to fine-tune the pre-trained language model can improve the model's accuracy.

[0063] Step S40: any of the second target sentences and the first target sentence are combined into a sentence pair, and the sentence pair is input into a pre-trained language model to generate a vector representation corresponding to the sentence pair.

[0064] It should be noted that in step S40, the vector representation corresponding to the sentence pair is a vector representation that fully contains the feature information of the second target sentence and the first target sentence. The pre-trained language model is a model that can generate a vector representation corresponding to the sentence pair when the sentence pair is input into the pre-trained language model. No detailed limitation is given here.

[0065] In one embodiment of the present invention, step S40 includes:

[0066] Step S41: input the sentence pair into a language model structure, process the sentence pair through the multi-head attention mechanism in the language model structure, and generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0067] Specifically, in step S41, the focus is on inputting the sentence pair into the language model structure and processing it with the help of the multi-head attention mechanism therein. The multi-head attention mechanism is an advanced and effective feature extraction and semantic association capture mechanism. When the sentence pair enters the language model structure, the multi-head attention mechanism begins to work. It will examine each word in the sentence pair (or more accurately, the word vector representation corresponding to the word) from multiple "heads" (that is, multiple different attention angles) at the same time and calculate the degree of association between them. For example, for a certain word in a sentence pair, different "heads" may pay attention to words that are related to it in grammatical structure, words that are semantically similar, words that are logically associated with it in the sentence pair, and so on. Through the above-mentioned processing process of the multi-head attention mechanism, it will adjust, weight, and other operations on the word vector of each position in the sentence pair, so that each word vector incorporates relevant semantic information from other parts of the sentence pair. Finally, all the processed word vectors in the entire sentence pair are arranged in order to form a feature representation sequence that integrates the semantic information of the sentence pair. This sequence is no longer a simple sequence of raw word vectors, but rather the result of fully considering the semantic interactions within the sentence pair. The vector at each position contains richer and more comprehensive semantic content, encompassing both its own original semantics and the associated semantics with other words in the context of the entire sentence pair.

[0068] In step S41, the multi-head attention mechanism focuses on the semantic relationships within a sentence pair from multiple different perspectives. Compared to traditional single-head attention methods, this can more comprehensively capture the complex semantic connections between sentences. For example, a sentence pair may have both grammatical dependencies and deep semantic logical connections. Different "heads" can focus on these different aspects to ensure that important semantic information is not missed. In natural language, words sometimes have long-range semantic dependencies. For example, in a long sentence or complex sentence pair, the subject and object may be far apart, but they still have important semantic connections. The multi-head attention mechanism can bridge these distances, accurately find and utilize this long-range semantic dependency, so that the generated feature representation sequence can fully reflect the semantic picture of the sentence pair. The feature representation sequence generated by the multi-head attention mechanism incorporates multi-dimensional semantic information at each position. This enables subsequent tasks based on this feature representation (such as classification, similarity calculation, etc.) to utilize richer and more accurate semantic features, avoiding the one-sidedness that may result from using only a single dimension of semantic information, thereby improving the accuracy and effectiveness of the tasks. Because it incorporates semantic information from all parts of a sentence pair, the feature representation is more resistant to noise, local semantic ambiguity, and other situations. Even if the semantic expression of a part of a sentence pair is slightly ambiguous or there are some interference factors, the overall semantic fusion can still extract relatively accurate and stable semantic features, which is conducive to the stable implementation of subsequent tasks. The feature representation sequence generated by the multi-head attention mechanism can be easily applied to a variety of different natural language processing tasks because of its high-quality and comprehensive capture of the semantic information of the sentence pair. Whether it is determining the category of a sentence pair in text classification tasks, measuring the similarity between sentence pairs in semantic similarity tasks, or mining the entity relationships contained in sentence pairs in relationship extraction tasks, this feature representation can provide strong semantic support without the need for significant feature adjustments or reconstruction for different tasks.

[0069] In one embodiment of the present invention, step S41 includes:

[0070] Step S411: input the sentence pair into a Transformer-based Cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0071] Understandably, the Transformer architecture has performed exceptionally well in natural language processing. The Transformer-based cross-encoder is highly effective at processing sequential data, specifically text sequences such as natural language sentence pairs. Its self-attention mechanism effectively captures semantic relationships within and between sentences. Compared to traditional neural network architectures, the Transformer is better able to handle long sequences of data, avoiding the vanishing and exploding gradient problems associated with long-sequence information transfer. It also enables parallel computation, improving computational efficiency. The cross-encoder is specifically designed to encode sentence pairs, and its architectural design considers how to best integrate the semantic information between the two sentences. When a sentence pair is input, it fully utilizes its internal structure and parameters to explore the grammatical, semantic, and logical connections between the two sentences in the pair, generating a high-quality feature representation sequence that incorporates this semantic information. In the cross-encoder, the two sentences in a sentence pair are not simply concatenated or processed separately; instead, the model engages in deep semantic interaction within the model. It can automatically learn the semantic correspondences between sentences. For example, for a question-answer pair, it can accurately capture the semantic matching points between the question and the answer. For sentence pairs that are being judged for semantic similarity, it can identify the key semantic components that reflect the similarity. This deep semantic interaction can generate a feature representation sequence that better reflects the overall semantics of the sentence pair, making it more beneficial for subsequent natural language processing tasks than simple concatenation or shallow fusion methods.

[0072] The cross-encoder accurately extracts semantic features from sentence pairs. By performing multi-level encoding and feature extraction on sentence pairs, it can focus on representative semantic elements within the sentence pairs, such as relationships between entities, actions, and attributes. The resulting feature representation sequence not only captures the overall semantic information of the sentence pair but also highlights key semantic features, providing valuable information for subsequent tasks such as text classification and semantic matching.

[0073] The Transformer-based cross-encoder has demonstrated excellent generalization capabilities across many natural language processing tasks. Whether it's text classification, semantic similarity calculation, information retrieval, or relation extraction, the generated feature representation sequence, incorporating semantic information, can serve as effective input features for further processing. This versatility eliminates the need for large-scale architectural adjustments across diverse downstream tasks, requiring only fine-tuning tailored to the task objectives.

[0074] The Cross-encoder can adapt relatively flexibly to different task requirements and data characteristics. For example, when the task focus shifts from semantic similarity judgment to relationship extraction, it can adapt the generated feature representation sequence to meet the new task requirements by adjusting model parameters or combining different loss functions, because it has already comprehensively and deeply mined the semantic information of the sentence pair.

[0075] Step S50 : determining a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair.

[0076] It should be noted that in step S50, the method of determining the target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector features corresponding to the sentence pair can be any feasible method. For example, the cosine similarity between the second target sentence and the first target sentence can be calculated based on the vector representation corresponding to the sentence pair.

[0077] In one embodiment of the present invention, step S50 includes:

[0078] Step S51, extracting the [CLS] representation in the vector representation;

[0079] Step S52: Input the [CLS] representation into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

[0080] It can be understood that the [CLS] representation is a comprehensive representation of the overall semantics of a sentence pair. Extracting it captures the most critical semantic information of the sentence pair, avoiding potential local semantic interference from other position vectors. The fully connected layer further nonlinearly transforms the extracted [CLS] representation, mapping it to a suitable space. This yields a matching score that is more consistent with the semantic match assessment and enhances the accuracy of similarity judgments.

[0081] Step S60: sorting the target similarities between each of the second target sentences and the first target sentences.

[0082] Step S70 : determining the weight ranking of each of the first words according to the ranking of each of the target similarities, wherein the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0083] In this way, when a sentence pair formed by two sentences is input into the pre-trained language model, a vector representation of the sentence pair can be obtained, and then the similarity of the two sentences can be obtained, and finally the similarity of each second target sentence to the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first word removed from the first target sentence in the second target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first word removed from the first target sentence in the second target sentence. In this way, the similarity of each second target sentence to the first target sentence is sorted, and the reverse order of the sorting obtained is the importance sorting of each first word in the first target sentence. In this way, since this scheme is a supervised learning model, it can avoid the problem of low word weight sorting accuracy caused by unsupervised methods in the prior art. In addition, this scheme does not require a large amount of training data like supervised methods in the prior art, which requires the defect of labeling weights of each word in the text one by one. Therefore, this scheme can avoid the problem of supervised methods in the prior art that labeling data is time-consuming and difficult to obtain. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted.

[0084] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] In one embodiment, a word weight ranking device is provided, which corresponds to the word weight ranking method in the above embodiment. Figure 4 As shown, the word weight ranking device includes an acquisition module 410, a removal module 420, a generation module 430, a first determination module 440, a ranking module 450, and a second determination module 460. The functional modules are described in detail as follows:

[0086] An acquisition module 410 is configured to acquire a first target sentence and segment the first target sentence to obtain a first vocabulary contained in the first target sentence;

[0087] a removal module 420 configured to remove each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words;

[0088] A generation module 430 is configured to combine any second target sentence and the first target sentence into a sentence pair, input the sentence pair into a pre-trained language model, and generate a vector representation corresponding to the sentence pair;

[0089] A first determination module 440 is configured to determine a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair;

[0090] A sorting module 450 is configured to sort the target similarities between each of the second target sentences and the first target sentence;

[0091] The second determining module 460 is configured to determine a weight ranking of each of the first words according to the ranking of each of the target similarities, wherein the weight ranking of the first words is opposite to the ranking of each of the target similarities.

[0092] In one embodiment, the removal module 420 is further configured to:

[0093] Get the first statement;

[0094] generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold;

[0095] Forming a first triplet of data from the first sentence, the second sentence, and the third sentence, and inputting the first triplet of data into a language model to be trained, and training the language model to be trained;

[0096] Calculate the output value of the loss function of the language model to be trained during the training process until the output value of the loss function reaches a preset value, and determine the language model to be trained as a pre-trained language model.

[0097] In one embodiment, the removal module 420 is further configured to:

[0098] Label the word weight data of the fourth sentence;

[0099] Extracting words in the fourth sentence whose word weight data is greater than a third preset threshold to obtain a fifth sentence;

[0100] Extracting words whose word weight data is less than a fourth preset threshold from the fourth sentence to obtain a sixth sentence;

[0101] The fourth sentence, the fifth sentence and the sixth sentence are formed into a second triplet data, and the second triplet data is input into the pre-trained language model as fine-tuning data, and the pre-trained language model is fine-tuned to obtain a fine-tuned pre-trained language model.

[0102] In one embodiment, the generating module 430 is specifically configured to:

[0103] The sentence pair is input into a language model structure, and the sentence pair is processed through a multi-head attention mechanism in the language model structure to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0104] In one embodiment, the generating module 430 is further configured to:

[0105] The sentence pair is input into a Transformer-based cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

[0106] In one embodiment, the first determining module 440 is specifically configured to:

[0107] Extracting a [CLS] representation from the vector representation;

[0108] The [CLS] representation is input into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

[0109] In one embodiment, the loss function is:

[0110] L=max(s(query,pos)-s(query,neg)+margin,0);

[0111] Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

[0112] The present invention provides a device for ranking word weights. When a sentence pair consisting of two sentences is input into a pre-trained language model, a vector representation of the sentence pair can be obtained, and then the similarity between the two sentences can be obtained. Ultimately, the similarity between each second target sentence and the first target sentence can be obtained. The higher the similarity between the second target sentence and the first target sentence, the higher the importance of the first word removed from the second target sentence from the first target sentence. Similarly, the lower the similarity between the second target sentence and the first target sentence, the lower the importance of the first word removed from the second target sentence from the first target sentence. In this way, the similarity between each second target sentence and the first target sentence is ranked, and the reverse order of the ranking is the importance ranking of each first word in the first target sentence. In this way, because this scheme is a supervised learning model, it can avoid the problem of low word weight ranking accuracy caused by unsupervised methods in the prior art. Moreover, this scheme does not require a large amount of training data like supervised methods in the prior art, which requires the weight of each word in the text to be labeled one by one. Therefore, this scheme can avoid the problem of supervised methods in the prior art that labeling data is time-consuming and difficult to obtain. In summary, this solution can solve the problem in the prior art that word weights cannot be accurately sorted.

[0113] For the specific definition of the word weight sorting device, please refer to the definition of the word weight sorting method above, which will not be repeated here. The various modules in the above-mentioned word weight sorting device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] The ranking method based on the above word weights, such as Figure 5 As shown, an embodiment of the present invention further provides a schematic structural diagram of a word weight ranking device, which includes a processor 51 and a memory 52 coupled to the processor 51. The memory 52 stores a computer program, which, when executed by the processor 51, causes the processor 51 to perform the steps of the word weight ranking method in the above embodiment.

[0115] For other details about how the processor 51 in the above-mentioned word weight sorting device implements the above-mentioned technical solution, please refer to the description of the word weight sorting method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0116] Among them, the processor 51 can also be called a CPU (Central Processing Unit), and the processor 51 may be an integrated circuit chip with signal processing capabilities; the processor 51 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 51 can also be any conventional processor, etc.

[0117] like Figure 6 As shown, an embodiment of the present invention further provides a schematic structural diagram of a computer-readable storage medium, on which a readable computer program 61 is stored; wherein the computer program 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), and other media that can store program code, or a terminal device such as a computer, server, mobile phone, or tablet.

[0118] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0120] In addition, the functional modules in the various embodiments of the present invention may be integrated into a single processing module, each module may exist physically separately, or two or more modules may be integrated into a single module. The integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0121] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0122] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium), or a semiconductor medium (e.g., an SSD (solid state disk)).

[0123] The technical solution provided by the present invention is introduced in detail above. Specific examples are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0124] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, to the extent such modifications and variations fall within the scope of the claims and their equivalents, the present invention is intended to encompass such modifications and variations. Any software tools or components not provided by our company that appear in the embodiments of the present invention are for illustrative purposes only and do not represent actual use.

Claims

1. A method for ranking word weights, characterized in that: include: Acquire a first target sentence, segment the first target sentence into words, and obtain a first vocabulary set included in the first target sentence; removing each first word in the first vocabulary set from the first target sentence to obtain a second target sentence, wherein the number of the second target sentences is the same as the number of the first words, and the number of the first words contained in the second target sentence is one less than the total number of the first words; Combining any second target sentence with the first target sentence into a sentence pair, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair; determining a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair; sorting the target similarities between each of the second target sentences and the first target sentences; The weight ranking of each of the first words is determined according to the ranking of each of the target similarities, and the weight ranking of the first words is opposite to the ranking of each of the target similarities.

2. The word weight ranking method according to claim 1, characterized in that: After removing a random first word from the first vocabulary set from the first target sentence to obtain a second target sentence, until the number of the obtained second target sentences is equal to the number of the first vocabulary, and before forming a sentence pair with any second target sentence and the first target sentence, and inputting the sentence pair into a pre-trained language model to generate a vector representation corresponding to the sentence pair, the method further includes: Get the first statement; generating a second sentence having a semantic similarity with the first sentence higher than a first preset threshold, and generating a third sentence having a semantic similarity with the first sentence lower than a second preset threshold; Forming a first triplet of data from the first sentence, the second sentence, and the third sentence, and inputting the first triplet of data into a language model to be trained, and training the language model to be trained; Calculate the output value of the loss function of the language model to be trained during the training process until the output value of the loss function reaches a preset value, and determine the language model to be trained as a pre-trained language model.

3. The word weight ranking method according to claim 2, characterized in that: After the output value of the loss function reaches a preset value and the language model to be trained is determined as a pre-trained language model, the method further includes: Label the word weight data of the fourth sentence; Extracting words in the fourth sentence whose word weight data is greater than a third preset threshold to obtain a fifth sentence; Extracting words whose word weight data is less than a fourth preset threshold from the fourth sentence to obtain a sixth sentence; The fourth sentence, the fifth sentence and the sixth sentence are formed into a second triplet data, and the second triplet data is input into the pre-trained language model as fine-tuning data, and the pre-trained language model is fine-tuned to obtain a fine-tuned pre-trained language model.

4. The method for ranking word weights according to claim 1, wherein: The step of forming a sentence pair from any second target sentence and the first target sentence, inputting the sentence pair into a pre-trained language model, and generating a vector representation corresponding to the sentence pair includes: The sentence pair is input into a language model structure, and the sentence pair is processed through a multi-head attention mechanism in the language model structure to generate a feature representation sequence that integrates the semantic information of the sentence pair.

5. The word weight ranking method according to claim 4, characterized in that: The step of inputting the sentence pair into a language model structure, processing the sentence pair through a multi-head attention mechanism in the language model structure, and generating a feature representation sequence that integrates the semantic information of the sentence pair includes: The sentence pair is input into a Transformer-based cross-encoder to generate a feature representation sequence that integrates the semantic information of the sentence pair.

6. The method for ranking word weights according to claim 5, characterized in that: The determining, based on the vector representation corresponding to the sentence pair, a target similarity between the second target sentence and the first target sentence in the sentence pair, includes: Extracting a [CLS] representation from the vector representation; The [CLS] representation is input into a fully connected layer to obtain a matching score between the second target sentence and the first target sentence in the sentence pair.

7. The word weight ranking method according to claim 2, characterized in that: The loss function is: L=max(s(query,pos)-s(query,neg)+margin,0); Among them, L is the output value of the loss function, s(query,pos) is the similarity between query and pos, s(query,neg) is the similarity between query and neg, query is the first statement, pos is the second statement, neg is the third statement, and margin is a preset hyperparameter.

8. A word weight sorting device, characterized in that: include: an acquisition module, configured to acquire a first target sentence, and segment the first target sentence to obtain a first vocabulary contained in the first target sentence; a removal module, configured to remove each first word from the first target sentence, respectively, to obtain a second target sentence corresponding to each first word; a generation module, configured to combine any second target sentence and the first target sentence into a sentence pair, input the sentence pair into a pre-trained language model, and generate a vector representation corresponding to the sentence pair; a first determining module, configured to determine a target similarity between the second target sentence and the first target sentence in the sentence pair based on the vector representation corresponding to the sentence pair; a sorting module, configured to sort the target similarities between each of the second target sentences and the first target sentence; The second determining module is configured to determine a weight ranking of each of the first words according to the ranking of each of the target similarities, wherein the weight ranking of the first words is opposite to the ranking of each of the target similarities.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the word weight ranking method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the word weight ranking method according to any one of claims 1 to 7 are implemented.

Images