Sentence vector model optimization method, text similarity calculation method and device

Abstract

The application relates to the technical field of artificial intelligence sentence vector, and proposes a sentence vector model optimization method, a text similarity calculation method and device, an electronic equipment and a storage medium. The method comprises the following steps: determining a first sentence vector model according to a contrast learning loss function; wherein the first sentence vector model comprises a plurality of multi-head attention modules and a plurality of feedforward neural network modules; calculating the importance score of a first network parameter in each multi-head attention module and the importance score of a second network parameter in each feedforward neural network module; performing network parameter pruning on the first sentence vector model according to the importance score of the first network parameter and the importance score of the second network parameter, obtaining a second sentence vector model; and performing model retraining on the second sentence vector model to obtain an optimal sentence vector model, thereby improving the semantic expression effect of the optimal sentence vector model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence sentence vector technology, and in particular to a method for optimizing a sentence vector model, a method for calculating text similarity, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] In the field of artificial intelligence, sentence vectorization falls under the text representation domain of natural language processing. The goal of text representation is to convert text data into vector form that computers can understand and process, enabling various natural language processing tasks such as sentiment analysis, text classification, and question-answering systems. Sentence vectorization is a text representation method that maps the entire sentence to a fixed-dimensional vector space, capturing the sentence's semantic information and contextual relationships, thus facilitating subsequent machine learning or deep learning processing.

[0003] However, in current sentence vector models, there is a phenomenon where network parameters contribute unevenly or even harmfully to the semantic expression effect of the sentence vector model, resulting in a decrease in the semantic expression effect of the sentence vector model. Summary of the Invention

[0004] This application provides an optimization method for sentence vector models, a text similarity calculation method, an apparatus, an electronic device, and a storage medium, which can improve the semantic expression effect of sentence vector models. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an optimization method for a sentence vector model, comprising the following steps:

[0006] The first sentence vector model is determined based on the contrastive learning loss function; the first sentence vector model includes several multi-head attention modules and several feedforward neural network modules.

[0007] Calculate the importance score of the first network parameter in each multi-head attention module and the importance score of the second network parameter in each feedforward neural network module;

[0008] Based on the importance scores of the first network parameter and the second network parameter, the network parameters of the first sentence vector model are trimmed to obtain the second sentence vector model;

[0009] The second sentence vector model is retrained to obtain the optimal sentence vector model.

[0010] Secondly, embodiments of this application provide a text similarity calculation method, which includes the following steps:

[0011] Obtain the text corpus; the text corpus includes several texts;

[0012] Each text is input into the optimal sentence vector model obtained by the optimization method described above, and the corresponding sentence vector representation is obtained.

[0013] Calculate the similarity between several texts based on sentence vector representation.

[0014] Thirdly, embodiments of this application provide an optimization apparatus for a sentence vector model, comprising:

[0015] The first sentence vector model determination module is used to determine the first sentence vector model based on the contrastive learning loss function; the first sentence vector model includes several multi-head attention modules and several feedforward neural network modules.

[0016] The importance score calculation module is used to calculate the importance score of the first network parameter in each multi-head attention module and the importance score of the second network parameter in each feedforward neural network module;

[0017] The second sentence vector model acquisition module is used to perform network parameter pruning on the first sentence vector model based on the importance scores of the first network parameters and the second network parameters to obtain the second sentence vector model.

[0018] The optimal sentence vector model acquisition module is used to retrain the second sentence vector model to obtain the optimal sentence vector model.

[0019] Fourthly, embodiments of this application provide a text similarity calculation apparatus, including:

[0020] The text corpus acquisition module is used to acquire text corpora; the text corpus includes several texts.

[0021] The sentence vector representation acquisition module is used to input each text into the optimal sentence vector model obtained according to the above-mentioned sentence vector model optimization method, and obtain the corresponding sentence vector representation;

[0022] The similarity calculation module is used to calculate the similarity between several texts based on sentence vector representations.

[0023] Fifthly, embodiments of this application provide an electronic device, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first or second aspect.

[0024] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first or second aspect.

[0025] This application's embodiments determine a first sentence vector model based on a contrastive learning loss function. The first sentence vector model includes several multi-head attention modules and several feedforward neural network modules. The importance scores of the first network parameters in each multi-head attention module and the importance scores of the second network parameters in each feedforward neural network module are calculated. Based on the importance scores of the first and second network parameters, the network parameters of the first sentence vector model are pruned to obtain a second sentence vector model. The second sentence vector model is then retrained to obtain an optimal sentence vector model. This application improves the semantic expression performance of the optimal sentence vector model by pruning the network parameters of the first sentence vector model, filtering out harmful and low-contribution network parameters, obtaining a second sentence vector model, and retraining the second sentence vector model to obtain an optimal sentence vector model.

[0026] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings. Attached Figure Description

[0027] Figure 1 A flowchart illustrating an optimization method for a sentence vector model provided in one embodiment of this application;

[0028] Figure 2 A flowchart illustrating a text similarity calculation method provided in one embodiment of this application;

[0029] Figure 3 A schematic diagram of the structure of an optimization device for a sentence vector model provided in one embodiment of this application;

[0030] Figure 4 A schematic diagram of the structure of a text similarity calculation device provided in one embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0034] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0035] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0036] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0037] Please see Figure 1 , Figure 1 A flowchart illustrating the optimization method for the sentence vector model provided in this application embodiment is shown. The method includes the following steps:

[0038] S10: Determine the first sentence vector model based on the contrastive learning loss function; the first sentence vector model includes several multi-head attention modules and several feedforward neural network modules.

[0039] Among them, the contrastive learning technique can make semantically similar sentence vectors move closer to each other in the vector space, while making semantically dissimilar sentence vectors move further apart in the vector space.

[0040] Among them, the Multi-Head Attention (MHA) module can extend the attention mechanism to multiple heads, thereby enhancing the model's attention to different features. The Feedforward Neural Network (FNN) module is a unidirectional multi-layer network structure. The information flow of the network is unidirectional, from the input layer through a series of hidden layers, and finally to the output layer.

[0041] In this embodiment, a contrastive learning loss function can be constructed, and the pre-trained language model can be fine-tuned based on the contrastive learning loss function to obtain a first sentence vector model. The pre-trained language model includes, but is not limited to, Bert-base, Bert-large, and Roberta-base models. The pre-trained language model includes several multi-head attention modules and several feedforward neural network modules. The first sentence vector model obtained after fine-tuning the pre-trained language model also includes several multi-head attention modules and several feedforward neural network modules.

[0042] S20: Calculate the importance score of the first network parameter in each multi-head attention module and the importance score of the second network parameter in each feedforward neural network module.

[0043] The importance score of the first network parameter is used to evaluate the importance of the first network parameter in the multi-head attention module to the semantic expression effect of the first sentence vector model, and the importance score of the second network parameter is used to evaluate the importance of the second network parameter in the feedforward neural network module to the semantic expression effect of the first sentence vector model.

[0044] Specifically, the semantic representation effect of the first sentence vector model needs to meet the evaluation criteria in two major aspects: alignment and uniformity. Alignment refers to the high degree of correlation between sentence vectors with similar semantics in the vector space, while uniformity refers to the even distribution of all sentence vectors in the vector space.

[0045] In the embodiments of this application, an objective function that integrates alignment and uniformity can be constructed. Based on the objective function, the importance score of the first network parameter in each multi-head attention module and the importance score of the second network parameter in each feedforward neural network module are calculated.

[0046] S30: Based on the importance scores of the first network parameter and the second network parameter, perform network parameter pruning on the first sentence vector model to obtain the second sentence vector model.

[0047] In this embodiment, the first network parameters and the second network parameters with lower importance scores can be pruned to obtain the second sentence vector model.

[0048] S40: Retrain the second sentence vector model to obtain the optimal sentence vector model.

[0049] In this embodiment, the second sentence vector model can be retrained based on an unsupervised dataset and a contrastive learning loss function to obtain the optimal sentence vector model. Specifically, the contrastive learning loss function is minimized, and the second sentence vector model is iteratively trained multiple times using stochastic gradient descent until the preset number of training rounds is reached to obtain the optimal sentence vector model.

[0050] Applying the embodiments of this application, a first sentence vector model is determined based on the contrastive learning loss function. This first sentence vector model includes several multi-head attention modules and several feedforward neural network modules. The importance scores of the first network parameters in each multi-head attention module and the importance scores of the second network parameters in each feedforward neural network module are calculated. Based on the importance scores of the first and second network parameters, the network parameters of the first sentence vector model are pruned to obtain a second sentence vector model. The second sentence vector model is then retrained to obtain the optimal sentence vector model. This application, by pruning the network parameters of the first sentence vector model, filters out harmful and low-contribution network parameters to obtain the second sentence vector model. Retraining the second sentence vector model yields the optimal sentence vector model, thus improving the semantic expression effect of the optimal sentence vector model.

[0051] In an optional embodiment, step S10 includes steps S101 to S104, as follows:

[0052] S101: Obtain the first sample dataset; the first sample dataset includes the training set and the validation set.

[0053] In this embodiment, a sampled unsupervised dataset from Wikipedia is used as the training set in the first sample dataset. The senteval toolkit is obtained, which includes test frameworks and datasets for semantic text similarity tasks and transfer learning tasks. Data from the semantic text similarity task is used as the validation set for the first sample dataset.

[0054] S102: Input the sample data in the training set into the sentence vector model to be trained to obtain the first sentence vector representation; wherein, the first sentence vector representation includes the first positive sentence vector pair and the negative sentence vector pair.

[0055] The sentence vector model to be trained is a pre-trained language model. Pre-trained language models include, but are not limited to, Bert-base, Bert-large, and Roberta-base models.

[0056] In this embodiment, batch data is constructed on the training set and input into the embedding and dropout layers of the pre-trained language model to obtain first positive sentence vector pairs and negative sentence vector pairs. The batch data includes positive sample pairs and negative sample pairs, and the first positive sentence vector pair... For positive sample pairs (x) i x i + The sentence vector representation of ) and the negative sentence vector pair For negative sample pairs (x) i x j + The sentence vector representation of ).

[0057] S103: Construct a contrastive learning loss function based on the first positive sentence vector pair and the negative sentence vector pair.

[0058] In this embodiment, the formula for calculating the contrastive learning loss function is as follows:

[0059]

[0060] Where sim(h1, h2) represents the cosine similarity, which is calculated as follows: This is used to calculate the similarity between vectors. τ is the temperature coefficient, a hyperparameter in the contrastive learning framework, and n represents the number of samples.

[0061] S104: Based on the contrastive learning loss function, the sentence vector model to be trained is subjected to multiple rounds of iterative training. After each round of training, the training effect of the trained sentence vector model is evaluated through the validation set until the training rounds reach the preset number. The sentence vector model with the best training effect is then used as the first sentence vector model.

[0062] In this embodiment, the contrastive learning loss function is minimized, and stochastic gradient descent is used to iteratively train the sentence vector model to be trained through multiple rounds. During model training, after each round, the training effect of the trained sentence vector model is evaluated using validation set data. The semantic text similarity task serving as the validation set comprises seven sub-tasks, all evaluated using the Spearman correlation coefficient, with the average Spearman correlation coefficient of the seven sub-tasks as the evaluation metric. After each round of validation, the sentence vector model with the best training performance is saved as the first sentence vector model.

[0063] Based on the first sample dataset and the contrastive learning loss function, the sentence vector model to be trained can be automatically and quickly fine-tuned to obtain the first sentence vector model.

[0064] In an optional embodiment, step S20 includes steps S201 to S207, as follows:

[0065] S201: Obtain the second sample dataset.

[0066] In this embodiment of the application, the senteval toolkit is loaded, wherein the senteval toolkit includes a dataset for a semantic text similarity task, and this dataset is used as a second sample dataset.

[0067] S202: Input the sample data of the second sample dataset into the first sentence vector model to obtain the second sentence vector representation; wherein, the second sentence vector representation includes the second positive sentence vector pair and the sentence vector representation of different sentences;

[0068] In this embodiment of the application, the sample data of the second sample dataset is input into the embedding layer and dropout layer of the first sentence vector model to obtain the second sentence vector representation.

[0069] S203: Construct the second loss function based on the second positive sentence vector pair.

[0070] In this embodiment, the second loss function is an alignment loss function. The closer the semantics of the sentence vector pairs, the lower the value of the alignment loss function. The alignment loss function is used to promote the relevance of sentence vectors with similar semantics in the vector space, that is, to improve the alignment of the sentence vector model. The formula for the second loss function is as follows:

[0071]

[0072] in, Let N represent the second loss function. pos The second positive sentence vector pair h represents the number of samples. p h p + A pair of positive samples x in the representative sample p x p + The sentence vector representation.

[0073] S204: Construct a third loss function based on the sentence vector representations of different sentences.

[0074] In this embodiment, the third loss function is a uniformity loss function. The lower the semantic relevance of sentence vector pairs, the lower the value of the uniformity loss function. The uniformity loss function is used to make sentence vectors with different semantics more distant, thereby making all sentence vectors evenly distributed in the vector space, i.e., improving the uniformity of the sentence vector model. The formula for the third loss function is as follows:

[0075]

[0076] in, This represents the third loss function. h represents the number of samples. p h q Different sentences x in the representative sample p x q The sentence vector representation.

[0077] S205: Perform a weighted summation of the second and third loss functions to obtain the target loss function.

[0078] In this embodiment, the formula for the target loss function is as follows:

[0079]

[0080] in, Let λ represent the target loss function, λ represent the weight coefficient of the second loss function, and 1-λ represent the weight coefficient of the third loss function.

[0081] Taking into account both alignment and uniformity, the target loss function is obtained by balancing their contributions to the sentence vector model through the weight coefficient λ.

[0082] S206: Perform masking processing on the first network parameters in each multi-head attention module to obtain the corresponding first mask variables; perform masking processing on the second network parameters in each feedforward neural network module to obtain the corresponding second mask variables.

[0083] Masking refers to setting a set of mask variables and associating these mask variables with the first network parameters of the multi-head attention module and the second network parameters of the feedforward neural network module.

[0084] In the embodiments of this application, the first network parameters and the second network parameters are respectively masked to obtain the first mask variable and the second mask variable.

[0085] S207: Based on the first mask variable and the target loss function, obtain the importance score of the first network parameter in each multi-head attention module; based on the second mask variable and the target loss function, obtain the importance score of the second network parameter in each feedforward neural network module.

[0086] In this embodiment, the formula for calculating the importance score of the first network parameter is as follows:

[0087]

[0088] in, This represents the importance score of the first network parameter in the i-th multi-head attention module. Let ξ represent the target loss function. (i)Let E represent the first mask variable corresponding to the first network parameters, and D represent the second sample dataset. This represents the partial derivative.

[0089] The formula for calculating the importance score of the second network parameter is as follows:

[0090]

[0091] in, ν represents the importance score of the second network parameter in the i-th feedforward neural network module. (i) This represents the second mask variable corresponding to the second network parameter.

[0092] By setting the first mask variable and the second mask variable, and combining them with the target loss function, the importance scores of the first network parameter and the second network parameter can be obtained automatically and quickly.

[0093] In an optional embodiment, step S30 includes steps S301 to S302, as follows:

[0094] S301: Sort the importance scores of the first network parameter and the second network parameter from high to low according to their importance scores;

[0095] S302: Prune the first and second network parameters, which are ranked lower, from the first sentence vector model to obtain the second sentence vector model.

[0096] The preset quantity can be manually set according to needs.

[0097] In this embodiment, the first network parameters and the second network parameters are arranged according to their importance scores, and a preset number of network parameters with lower importance scores are selected for pruning.

[0098] By pruning the network parameters of the first sentence vector model, the second sentence vector model can be obtained automatically and quickly.

[0099] In an optional embodiment, step S302 includes steps S3021 to S3023, as follows:

[0100] S3021: Obtain the network parameter clipping ratio.

[0101] The network parameter pruning ratio can be a preset value.

[0102] S3022: Determine the number of network parameters to be clipped based on the network parameter clipping ratio, and use the number of network parameters to be clipped as the preset number.

[0103] In this embodiment, the total number of first network parameters is obtained, and the product of the total number of first network parameters and the network parameter pruning ratio is taken as the number of first network parameters to be pruned. The product of the total number of second network parameters and the network parameter pruning ratio is taken as the number of second network parameters to be pruned. The sum of the number of first network parameters to be pruned and the number of second network parameters to be pruned is taken as a preset number.

[0104] S3023: Set the first mask variable corresponding to the first network parameter of the preset number that is sorted last and the first mask variable corresponding to the second network parameter to zero.

[0105] In this embodiment, the first mask variable corresponding to the first network parameter to be pruned is set to 0, and the second mask variable corresponding to the second network parameter to be pruned is set to 0, so that the first network parameter to be pruned and the second network parameter to be pruned no longer participate in the first sentence vector model operation.

[0106] By pruning the first and second network parameters that are ranked lower by a predetermined number, the second sentence vector model can be optimized to obtain the optimal sentence vector model.

[0107] Please see Figure 2 The image below is a flowchart illustrating a text similarity calculation method provided in one embodiment of this application. The method includes the following steps:

[0108] S100: Obtain the text corpus; the text corpus includes several texts.

[0109] Text similarity calculation is a common application problem in the field of natural language processing, aiming to identify whether two pieces of text are semantically similar. Text similarity calculation can be applied to scenarios such as deduplication of similar texts in text retrieval, matching user-input questions with standard template questions in intelligent question-answering systems, and similar text classification.

[0110] The text can be a single sentence or a paragraph consisting of multiple sentences.

[0111] In this embodiment, the text corpus can be determined by the specific application scenario. Specifically, in an intelligent question-answering scenario, the text corpus can be the user-inputted question and pre-stored standard questions in a question database.

[0112] S200: Input each text into the optimal sentence vector model obtained by the above sentence vector model optimization method to obtain the corresponding sentence vector representation.

[0113] In this embodiment, if the text is a single sentence, inputting the text into the optimal sentence vector model can obtain the sentence vector representation corresponding to the single sentence. If the text is a paragraph, the paragraph is divided into several sentences, and each sentence is input into the optimal sentence vector model to obtain the sentence vector representation corresponding to each sentence. Then, the sentence vector representations are concatenated to obtain the sentence vector representation corresponding to the paragraph.

[0114] S300: Calculate the similarity between several texts based on sentence vector representations.

[0115] In the embodiments of this application, the similarity between multiple texts can be calculated using cosine distance or Euclidean distance, and no limitation is made here.

[0116] In intelligent question answering scenarios, when an intelligent question answering system receives a question input by a user, it calculates the text similarity between the user's input question and the standard questions pre-stored in the question library, matches the target standard question that is closest to the input question from the question library, and then displays the answer corresponding to the target standard question to the user.

[0117] By using the optimal sentence vector model provided in this application embodiment to represent text with sentence vectors, the accuracy of text similarity calculation is improved, and high-precision matching between user-input questions and pre-stored standard questions in the question bank is achieved in intelligent question answering scenarios, thus meeting the user's need for rapid feedback in intelligent question answering scenarios.

[0118] Please see Figure 3 This is a schematic diagram of the structure of an optimization device for a sentence vector model provided in one embodiment of this application. This device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 5 includes:

[0119] The first sentence vector model determination module 51 is used to determine the first sentence vector model based on the contrastive learning loss function; wherein, the first sentence vector model includes several multi-head attention modules and several feedforward neural network modules;

[0120] Importance score calculation module 52 is used to calculate the importance score of the first network parameter in each multi-head attention module and the importance score of the second network parameter in each feedforward neural network module;

[0121] The second sentence vector model acquisition module 53 is used to perform network parameter pruning on the first sentence vector model based on the importance scores of the first network parameters and the importance scores of the second network parameters to obtain the second sentence vector model.

[0122] The optimal sentence vector model acquisition module 54 is used to retrain the second sentence vector model to obtain the optimal sentence vector model.

[0123] It should be noted that the sentence vector model optimization device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the sentence vector model optimization method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the sentence vector model optimization device and the sentence vector model optimization method provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0124] Please see Figure 4 This is a schematic diagram of the structure of a text similarity calculation device provided in one embodiment of this application. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device 6 includes:

[0125] The text corpus acquisition module 61 is used to acquire text corpus; the text corpus includes several texts.

[0126] The sentence vector representation acquisition module 62 is used to input each text into the optimal sentence vector model obtained according to the above-mentioned sentence vector model optimization method to obtain the corresponding sentence vector representation;

[0127] The similarity calculation module 63 is used to calculate the similarity between several texts based on the sentence vector representation.

[0128] Please see Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Figure 3 As shown, the electronic device 21 may include: a processor 210, a memory 211, and a computer program 212 stored in the memory 211 and capable of running on the processor 210, such as: an optimization program for sentence vector models or a text similarity calculation program; when the processor 210 executes the computer program 212, it implements the steps in the above embodiments.

[0129] The processor 210 may include one or more processing cores. The processor 210 connects to various parts within the computer device 21 using various interfaces and lines. It executes various functions of the computer device 21 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 211, and by accessing data in the memory 211. Optionally, the processor 210 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 210 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 210.

[0130] The memory 211 may include random access memory (RAM) or read-only memory. Optionally, the memory 211 may include a non-transitory computer-readable storage medium. The memory 211 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 211 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 211 may also be at least one storage device located remotely from the aforementioned processor 210.

[0131] This application also provides a computer storage medium that can store multiple instructions. These instructions are applicable to being loaded by a processor and executed by the method steps of the above embodiments. For details of the execution process, please refer to the specific description of the above embodiments, which will not be repeated here.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0139] This invention is not limited to the above-described embodiments. If any modifications or variations to this invention do not depart from the spirit and scope of this invention, and if such modifications and variations fall within the scope of the claims and equivalent technologies of this invention, then this invention also intends to include such modifications and variations.

Claims

1. An optimization method for a sentence vector model, characterized in that, Includes the following steps: The first sentence vector model is determined based on the contrastive learning loss function; wherein, the first sentence vector model includes several multi-head attention modules and several feedforward neural network modules; Calculate the importance score of the first network parameter in each of the multi-head attention modules and the importance score of the second network parameter in each of the feedforward neural network modules; Based on the importance scores of the first network parameter and the second network parameter, the network parameters of the first sentence vector model are trimmed to obtain the second sentence vector model; The second sentence vector model is retrained to obtain the optimal sentence vector model; The step of determining the first sentence vector model based on the contrastive learning loss function includes: Obtain the first sample dataset; the first sample dataset includes a training set and a validation set; The sample data in the training set is input into the sentence vector model to be trained to obtain the first sentence vector representation; wherein, the first sentence vector representation includes a first positive sentence vector pair and a negative sentence vector pair; The contrastive learning loss function is constructed based on the first positive sentence vector pair and the negative sentence vector pair; According to the contrastive learning loss function, the sentence vector model to be trained is subjected to multiple rounds of iterative training. After each round of training, the training effect of the trained sentence vector model is evaluated through the validation set until the training rounds reach the preset number. The sentence vector model with the best training effect is taken as the first sentence vector model. The step of calculating the importance score of the first network parameter in each of the multi-head attention modules and the importance score of the second network parameter in each of the feedforward neural network modules includes: Obtain the second sample dataset; The sample data of the second sample dataset is input into the first sentence vector model to obtain the second sentence vector representation; wherein, the second sentence vector representation includes the second positive sentence vector pair and the sentence vector representation of different sentences; Construct a second loss function based on the second positive sentence vector pair; Based on the sentence vector representations of the different sentences, a third loss function is constructed; The target loss function is obtained by weighted summation of the second loss function and the third loss function; The first network parameters in each of the multi-head attention modules are masked to obtain corresponding first mask variables; the second network parameters in each of the feedforward neural network modules are masked to obtain corresponding second mask variables. Based on the first mask variable and the target loss function, the importance score of the first network parameter in each multi-head attention module is obtained; based on the second mask variable and the target loss function, the importance score of the second network parameter in each feedforward neural network module is obtained.

2. The optimization method for the sentence vector model according to claim 1, characterized in that: The step of pruning the network parameters of the first sentence vector model based on the importance scores of the first network parameters and the second network parameters to obtain the second sentence vector model includes: Based on the importance scores, the importance scores of the first network parameter and the second network parameter are sorted from highest to lowest. The first and second network parameters, which are ranked lower, are pruned from the first sentence vector model to obtain the second sentence vector model.

3. The optimization method for the sentence vector model according to claim 2, characterized in that: The step of pruning a predetermined number of first network parameters and second network parameters that are ranked lower from the first sentence vector model to obtain the second sentence vector model includes: Obtain the cropping ratio from the network parameters; Based on the network parameter pruning ratio, determine the number of network parameters to be pruned, and use the number of network parameters to be pruned as a preset number; Set the first mask variable corresponding to the first network parameter of the preset number that is sorted last, and the first mask variable corresponding to the second network parameter to zero.

4. A method for calculating text similarity, characterized in that, Includes the following steps: Acquire a text corpus; the text corpus includes several texts; Each of the texts is input into the optimal sentence vector model obtained by the optimization method of the sentence vector model according to any one of claims 1 to 3, and the corresponding sentence vector representation is obtained; Based on the sentence vector representation, the similarity between several texts is calculated.

5. An optimization device for a sentence vector model, characterized in that, include: The first sentence vector model determination module is used to determine a first sentence vector model based on a contrastive learning loss function. The first sentence vector model includes several multi-head attention modules and several feedforward neural network modules. Determining the first sentence vector model based on the contrastive learning loss function includes: acquiring a first sample dataset; the first sample dataset includes a training set and a validation set; inputting the sample data from the training set into the sentence vector model to be trained to obtain a first sentence vector representation; wherein the first sentence vector representation includes a first positive sentence vector pair and a negative sentence vector pair; constructing the contrastive learning loss function based on the first positive sentence vector pair and the negative sentence vector pair; performing multiple rounds of iterative training on the sentence vector model to be trained based on the contrastive learning loss function; evaluating the training effect of the trained sentence vector model using the validation set after each round of training, until the training rounds reach a preset number, and selecting the sentence vector model with the best training effect as the first sentence vector model. An importance score calculation module is used to calculate the importance score of the first network parameter in each of the multi-head attention modules and the importance score of the second network parameter in each of the feedforward neural network modules. The calculation of the importance score of the first network parameter in each of the multi-head attention modules and the importance score of the second network parameter in each of the feedforward neural network modules includes: acquiring a second sample dataset; inputting the sample data of the second sample dataset into the first sentence vector model to obtain a second sentence vector representation; wherein the second sentence vector representation includes a second positive sentence vector pair and sentence vector representations of different sentences; constructing a second loss function based on the second positive sentence vector pair; and based on... The sentence vector representations of the different sentences are used to construct a third loss function; a weighted sum of the second loss function and the third loss function is performed to obtain a target loss function; the first network parameters in each multi-head attention module are masked to obtain corresponding first mask variables; the second network parameters in each feedforward neural network module are masked to obtain corresponding second mask variables; the importance score of the first network parameter in each multi-head attention module is obtained based on the first mask variables and the target loss function; and the importance score of the second network parameter in each feedforward neural network module is obtained based on the second mask variables and the target loss function. The second sentence vector model acquisition module is used to perform network parameter pruning on the first sentence vector model based on the importance scores of the first network parameters and the importance scores of the second network parameters to obtain the second sentence vector model. The optimal sentence vector model acquisition module is used to retrain the second sentence vector model to obtain the optimal sentence vector model.

6. A text similarity calculation device, characterized in that, include: The text corpus acquisition module is used to acquire text corpus; the text corpus includes several texts. The sentence vector representation acquisition module is used to input each of the texts into the optimal sentence vector model obtained by the optimization method of the sentence vector model according to any one of claims 1 to 3, and obtain the corresponding sentence vector representation; The similarity calculation module is used to calculate the similarity between several texts based on the sentence vector representation.

7. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images