Text vector model training method, text comparison method and system

Abstract

The invention discloses a text vector model training method and system and a text comparison method and system. The training method comprises the steps of obtaining a training text and processing the training text into an original lexical element sequence; the method comprises the following steps: generating a disturbed lexical element sequence by performing length disturbance on an original lexical element sequence, and constructing a homologous sample pair; the homologous sample pairs are input into a text vector model, the model extracts features through an encoder module, global vectors and memory vectors are generated through a global semantic module and a memory enhancement module respectively, and text representation vectors are obtained through a fusion module; using the vector of the homologous sample pair as a positive sample, and combining with a negative sample to calculate and compare learning loss so as to optimize model parameters until the model converges; according to the method, the bias dependence of the model on the text length is effectively inhibited, the distinguishing capability on a logic structure is enhanced, and the accuracy and robustness of text similarity comparison are improved.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a text vector model training method, a text comparison method, and a system. Background Technology

[0002] Text similarity comparison belongs to the field of natural language processing technology. It is usually achieved by encoding text into vector representations and calculating the similarity between vectors. At present, pre-trained language models with the Transformer architecture as the core have become the mainstream technology for obtaining text vectors. By extracting sentence vectors or text vectors through such models and combining them with training objectives such as contrastive learning to fine-tune the model in order to optimize the semantic embedding space, it constitutes the general method of current text similarity comparison.

[0003] However, in practical applications, existing general methods still have significant technical shortcomings. First, the model is susceptible to length bias. Since the Transformer architecture itself relies on positional encoding, when positive sample pairs in the training data have similar text lengths while negative sample pairs have significantly different lengths, the model may treat text length as a pseudo-feature for judging semantic similarity rather than truly understanding the semantic content. This leads to misjudgment of similar texts with large length differences during the inference stage, affecting the stability of threshold determination. Second, the text vectors obtained by common global pooling operations are static representations, which have limited ability to capture the logical structure information within the text. They are difficult to distinguish between text pairs with similar words but different logical relationships, thus causing semantic misjudgment. In addition, in contrastive learning training, if positive sample pairs are constructed solely by weak enhancement methods such as random deactivation, the view difference is limited, making it difficult for the model to learn deep semantic discriminative features, thus restricting its generalization ability in real-world scenarios.

[0004] Therefore, existing technologies suffer from problems such as low semantic discriminability and poor stability due to the limitations of length bias and static pooling strategies, as well as insufficient diversity of positive sample pairs. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a text vector model training method, a text comparison method and system to solve the problems of length bias interference, weak representation ability of sequence logical structure and insufficient model discriminability in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a text vector model training method, comprising: A. Obtain the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches; B. Perform a length-suppressed processing on the original word sequence of each training text in the current training batch to generate the corresponding perturbed word sequence; C. Based on the original word sequence and perturbed word sequence of each training text in the current training batch, construct positive sample pairs and negative sample pairs for each training text, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the pre-built text vector model. D. Repeat steps B to C until the preset model convergence condition is met, and then output the trained text vector model.

[0007] In an optional implementation, the original word sequence of each training text is subjected to a suppression length bias, specifically including the following steps: With a preset repetition probability, at least one word position is randomly selected from the original word sequence of each training text to determine the set of words to be repeated. For each lexical in the set of lexicals to be repeated, insert a copy of that lexical at its adjacent position to generate a perturbed lexical sequence for suppressing length bias; Determine whether the length of the perturbation word sequence exceeds the preset maximum sequence length: If the maximum sequence length is exceeded, the perturbation word sequence is truncated to meet the constraint of the maximum sequence length, and the truncated perturbation word sequence is output. If it does not exceed the limit, output the perturbation word sequence.

[0008] In an optional implementation, step C specifically includes the following steps: Based on the original word sequence and perturbed word sequence of each training text in the current training batch, construct positive sample pairs; Construct a negative sample set; The negative sample set includes the original word sequence and the perturbed word sequence corresponding to other training texts in the current training batch; Based on the original word sequence of each training text in the current training batch, construct negative sample pairs with the original word sequence or perturbed word sequence in the negative sample set; The original word sequence and the perturbation word sequence in the positive sample pair are respectively input into the pre-constructed text vector model to obtain the original text representation vector and the perturbation text representation vector, and the similarity of the positive sample pair is calculated based on the original text representation vector and the perturbation text representation vector. The original word sequence or perturbed word sequence in the negative sample pair is input into the text vector model to obtain the negative sample text representation vector, and the similarity of the negative sample pair is calculated based on the original text representation vector and the negative sample text representation vector. With the optimization objective of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, a contrastive learning loss function is used to calculate the loss value, and the parameters of the text vector model are updated based on the loss value through a backpropagation algorithm.

[0009] In an optional implementation, the original word sequence and the perturbation word sequence in the positive sample pair are respectively input into a pre-built text vector model to obtain the original text representation vector and the perturbation text representation vector. Based on the original text representation vector and the perturbation text representation vector, the similarity of the positive sample pair is calculated, specifically including the following steps: The original word sequence and the perturbed word sequence in the positive sample pair are respectively input into the pre-built text vector model; The text vector model includes an encoder module, a memory enhancement module, a global semantic module, and a fusion module. The inputs of the memory enhancement module and the global semantic module are connected to the output of the encoder module, and the outputs of the memory enhancement module and the global semantic module are connected to the input of the fusion module. Based on the encoder module, the original word sequence and the perturbed word sequence are encoded respectively to obtain the corresponding original sequence features and perturbed sequence features; Based on the global semantic module, pooling operations are performed on the original sequence features and the perturbed sequence features respectively to obtain the corresponding first global semantic vector and second global semantic vector; According to the memory enhancement module, the original sequence features and the perturbation sequence features are sequentially updated and aggregated to obtain the corresponding first memory vector and second memory vector; According to the fusion module, the first global semantic vector and the first memory vector are fused to obtain the original text representation vector; According to the fusion module, the second global semantic vector and the second memory vector are fused to obtain the perturbation text representation vector.

[0010] In one optional implementation, the memory enhancement module includes a memory update unit and a readout unit; The memory update unit is used to iteratively update the memory state of the current word according to the word order of the input sequence features, based on the features of the current word and the memory state of the previous word. The readout unit is used to generate output features based on the memory state corresponding to each word, and to perform pooling operations on the output features of all words to generate a memory vector.

[0011] In an optional implementation, the memory enhancement module further includes a preprocessing unit; The preprocessing unit is used to perform local aggregation operations on the sequence features before the sequence features are input into the memory update unit.

[0012] In an optional implementation, the negative sample set further includes the original text representation vector and the perturbed text representation vector corresponding to the training text in the historical training batches.

[0013] Secondly, the present invention provides a text similarity comparison method, comprising: Obtain the first and second texts to be compared; The first and second texts to be compared are respectively input into the text vector model trained by the text vector model training method described above, and the first text representation vector and the second text representation vector are output. Based on the similarity function, the similarity score between the first text representation vector and the second text representation vector is calculated.

[0014] Thirdly, the present invention provides a text vector model training system, comprising: The training set acquisition module is used to acquire the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches. The text processing module is used to perform suppression length bias processing on the original word sequence of each training text in the current training batch to generate the corresponding perturbation word sequence. The loss calculation module is used to construct positive and negative sample pairs for each training text based on the original and perturbed word sequences of each training text in the current training batch, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the pre-built text vector model. The model optimization module is used to repeatedly execute the text processing module and the loss calculation module until the preset model convergence condition is met, and then output the trained text vector model.

[0015] Fourthly, the present invention provides a text similarity comparison system, comprising: The text acquisition module is used to acquire the first and second texts to be compared. The vector generation module is equipped with a text vector model trained using the text vector model training method described above. It is used to input the first text and the second text to be compared into the text vector model respectively, and output the first text representation vector and the second text representation vector. The text comparison module is used to calculate the similarity score between the first text representation vector and the second text representation vector based on a similarity function.

[0016] The beneficial effects of the embodiments provided by the present invention include: This invention introduces a length perturbation mechanism for training texts to construct positive sample pairs with length differences. This forces the model to learn semantic features that are insensitive to changes in text length during training, effectively suppressing the model's dependence on surface features such as text length. This improves the robustness and consistency of the model in similarity comparisons between texts of different lengths.

[0017] This invention introduces a memory enhancement module containing a memory update unit and a readout unit into the text vector model to perform sequential state accumulation and aggregation on the sequence features of the text. This enables the generated text vector to not only contain overall semantic information, but also richer temporal and logical structure information, thereby enhancing the ability to distinguish text pairs with similar word meanings but different logical orders and solving the problem of logical misjudgment caused by static pooling.

[0018] This invention constructs a contrastive learning framework that includes diverse sources of negative examples, and optimizes model parameters with the goal of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. This effectively drives the model to learn more discriminative semantic features, ultimately improving the model's similarity discrimination accuracy and generalization ability in complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0020] Figure 1 A flowchart illustrating the text vector model training method in the embodiments of this specification is shown. Figure 2 An example diagram of the length bias suppression processing in the embodiments of this specification is shown; Figure 3 A schematic diagram of the contrastive learning framework for the text vector model training method in the embodiments of this specification is shown. Figure 4 A schematic diagram of the text vector model training system in the embodiments of this specification is shown; Figure 5 A schematic diagram of the text similarity comparison system in the embodiments of this specification is shown. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention.

[0022] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations are included in a specific order. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or may be performed in parallel. The operation numbers, such as S1, S2, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0023] Example 1

[0024] like Figure 1 As shown, this embodiment provides a text vector model training method, including: S1. Obtain the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches. In some embodiments, a training text set containing N training texts is first obtained, and normalization processing is performed on each training text in the set in sequence. Then, the normalized training text is segmented into words or sub-words to obtain the corresponding original word sequence. The processed original training text set is divided into training batches to obtain multiple training batches and corresponding original word sequences. The expression for the training text set is: ; In the formula, For the training text set; This represents the i-th training text sample, and N is the total number of training samples; The expression for the original lexical sequence is: W = [w1, w2, …, w n ]; In the formula, W is the original word sequence; w n Let n be the nth word in the sequence; n is the length of the word sequence. Specifically, standardization includes unifying the format of whitespace characters and punctuation marks, and filtering out non-standard control characters.

[0025] S2. Perform a length-suppression bias on the original word sequence of each training text in the current training batch to generate the corresponding perturbation word sequence. In some embodiments, the original word sequence of each training text is subjected to a suppressed length bias, specifically including the following steps: With a preset repetition probability, at least one word position is randomly selected from the original word sequence of each training text to determine the set of words to be repeated. For each lexical unit in the set of lexical units to be repeated, at least one copy of that lexical unit is inserted at its adjacent position to generate a perturbed lexical unit sequence for suppressing length bias. Preferably, to reduce the risk of semantic drift, protection rules are set to perform special processing on specific types of lexical units; For example, for numbers, dates, amounts, ID strings, negation words, and proper noun entity words, duplicate insertion or a reduction in their probability of being selected is not performed.

[0026] The maximum number of repetitions for a single word character shall not exceed the preset maximum number of repetitions. To avoid semantic drift or abnormal redundancy; Specifically, the perturbation strength of the perturbation sequence is controlled by the repetition probability; The functional expression for the repetition probability is: ; In the formula, Let be the probability of repetition, and The value is within the preset probability range; It is a perturbation word sequence; The length of the original word sequence; The length of the perturbation word sequence; Determine whether the length of the perturbation word sequence exceeds the preset maximum sequence length. : If the maximum sequence length is exceeded, the perturbation word sequence is truncated to meet the constraint of the maximum sequence length, and the truncated perturbation word sequence is output. If the length of the perturbation word sequence exceeds the preset maximum sequence length, the tail of the perturbation word sequence will be truncated or clipped using a sliding window. If it does not exceed the specified value, output the perturbation word sequence. In this embodiment, in order to adapt to the dimensionality requirements of the subsequent model input layer, the processing range of the word sequence is constrained by setting a maximum sequence length, so as to ensure the computational efficiency during batch training and avoid the loss of truncated information due to excessively long sequences.

[0027] like Figure 2As shown, and with a specific example, the process for suppressing length bias can be further explained as follows: Original text—This project conducted research on trains with a speed of 160 km / h based on the 25T type passenger car, and the original word sequence was input after word segmentation; First, the candidate annotation stage is entered, which automatically identifies and annotates specific types of words that need to be protected. For example, the numbers 25T and 160, as well as proper nouns such as bus, speed, kilometer, and train are selected and marked. These words will be excluded or have their selection probability reduced in subsequent repeated operations according to the protection rules. Subsequently, the sampling control phase begins. Based on the preset repetition probability and the maximum number of repetitions of a word, random sampling is performed from the unprotected word set. For the selected word, one or more copies are inserted in its adjacent positions, thereby generating a perturbed word sequence with increased length and altered local structure. Next, the process enters the length correction stage, where the system determines whether the length of the current perturbation word sequence exceeds the preset maximum sequence length. If the number of perturbations does not exceed the limit, the perturbation word sequence is output directly. If the length exceeds the limit, a truncation operation is performed to satisfy the length constraint before outputting the corresponding perturbation word sequence.

[0028] S3. Based on the homologous sample pairs of each training text in the current training batch, construct positive sample pairs and negative sample pairs for each training text, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the text vector model.

[0029] like Figure 3 As shown, exemplarily, step S3 specifically includes the following steps: S301. Construct positive sample pairs using the original lexical sequence and the perturbed lexical sequence corresponding to each training text in the current training batch. ; S302, Constructing a negative sample set ; The negative sample set includes the original word sequence and the perturbed word sequence corresponding to other training texts in the current training batch; In some embodiments, to expand the coverage of negative samples, the negative sample set also includes the original word sequence and perturbed word sequence corresponding to all training texts in the historical training batches; Among them, the word sequence corresponding to all training texts in the historical training batch is stored in a queued negative example library; Optionally, to further improve training stability, the queued negative example library is a momentum-updated queued negative example library, which generates the original word sequence and perturbation word sequence in the queued negative example library through an auxiliary encoder; The auxiliary encoder has the same structure as the main encoder but its parameters are independent. Its parameters are updated slowly through momentum updates to make the word sequence generated by the auxiliary encoder smoother and more stable, thus providing more consistent and high-quality negative samples for contrastive learning and effectively promoting robust training of the model.

[0030] S303. Based on the original word sequence of each training text in the current training batch, construct a negative sample pair with any original word sequence or perturbed word sequence in the negative sample set. ; S304. Input the original word sequence and the perturbation word sequence in the positive sample pair into the pre-constructed text vector model to obtain the original text representation vector and the perturbation text representation vector, and calculate the similarity of the positive sample pair based on the original text representation vector and the perturbation text representation vector. The text vector model includes an encoder module, a memory enhancement module, a global semantic module, and a fusion module. The inputs of the memory enhancement module and the global semantic module are connected to the output of the encoder module, and the outputs of the memory enhancement module and the global semantic module are connected to the input of the fusion module. For example, step S304 specifically includes the following steps: S3041. Combine the original word sequence W and the perturbed word sequence in the positive sample pair. The corresponding original sequence features are then input into the encoder module for encoding. and perturbation sequence features ; Specifically, the original word sequence and the perturbed word sequence are vectorized and encoded using the same encoder module that shares the same set of parameters; the encoder module can be a Transformer encoder or a variant of the Transformer encoder; The expression for the original sequence features is: ; In the formula, The original sequence features of the i-th training sample; is the latent vector of the b-th word in the original word sequence; b is the length of the original word sequence; The expression for the perturbation sequence characteristics is as follows: ; In the formula, The perturbation sequence features of the i-th training sample; is the latent vector of the b-th word in the perturbation word sequence; The length of the perturbation word sequence; S3042. Based on the global semantic module, analyze the original sequence features respectively. and perturbation sequence features Perform global pooling to obtain the corresponding first global semantic vector. Second global semantic vector ; In some embodiments, the global pooling operation may employ any combination of one or more of the following: first-word metavector extraction, average pooling, and weighted pooling. S3043. Based on the memory enhancement module, respectively, the original sequence features are processed... and the perturbation sequence features Perform sequential updates and aggregations to obtain the corresponding first memory vector. Second memory vector ; In some embodiments, the memory enhancement module includes a preprocessing unit, a memory update unit, and a readout unit connected in sequence. The preprocessing unit is used to perform causal convolution operations or other sequentially consistent local aggregation operations on the sequence features before the sequence features are input into the memory update unit, so as to enhance the expression of short-range dependencies and local step structures, while avoiding the use of future position information; The memory update unit is used to iteratively update the memory state of the current word based on the word order of the input sequence features, the features of the current word, and the memory state of the previous word. ; The memory state of the current word is expressed as a function: ; In the formula, This represents the memory state of the current word element; This is a memory state update function used to write the current input information into memory and retain historical information; This refers to the memory state of the previous word; The readout unit is used to generate output features based on the memory state corresponding to each word. And pooling is performed on the output features of all words to generate memory vectors; The functional expression for the output feature is: ; In the formula, The output feature of the current word element; Function for reading out the memory state; S3044. According to the fusion module, the first global semantic vector and the first memory vector are fused to obtain the original text representation vector; S3045. According to the fusion module, the second global semantic vector and the second memory vector are fused to obtain the perturbation text representation vector.

[0031] The vector fusion method in the fusion module is not limited and can be any combination of one or more of the following: splicing mapping, weighted summation, gated fusion, and residual fusion. In this embodiment, the following residual fusion method is adopted: ; ; In the formula, The perturbation term introduced for the memory vector; and These are different linear mapping functions; The gating coefficient; The sigmoid function is used; m is the memory vector. This is a global semantic vector; The text representation vector output by the fusion module.

[0032] In this embodiment, multi-level feature extraction and fusion of text sequences are achieved through the collaborative work of the encoder module, memory enhancement module, global semantic module, and fusion module. The memory enhancement module captures the temporal logical structure of the text through sequentially updated memory states, the global semantic module extracts the overall semantic information of the text through pooling operations, and the fusion module combines global semantic information and temporal memory information complementaryly through an effective feature integration mechanism. This results in a text representation vector that retains global semantic consistency while enhancing the ability to distinguish logical differences in sequences, providing a more discriminative feature foundation for subsequent similarity calculations.

[0033] S305. Input the original word sequence or perturbed word sequence in the negative sample pair into the text vector model to obtain the negative sample text representation vector, and calculate the similarity of the negative sample pair based on the original text representation vector and the negative sample text representation vector.

[0034] S306. With the optimization objective of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the contrastive learning loss function is used to calculate the contrastive loss value, and the parameters of the text vector model are updated based on the loss value through the backpropagation algorithm. Specifically, based on the contrastive loss value, the parameters of the encoder module, memory enhancement module, global semantic module, and fusion module in the text vector model are updated using the backpropagation algorithm.

[0035] Specifically, the expression for the contrastive learning loss function is: ; In the formula, Let be the contrastive loss value for the i-th training sample; This is a similarity calculation function; To compare the temperature coefficient of loss; Let be the original text representation vector of the i-th training sample; Let be the perturbation text representation vector of the i-th training sample; Let be the negative sample text representation vector of the j-th training sample; In this embodiment, the contrastive learning loss function adopts a one-to-many contrastive form, that is, one positive sample pair is compared with multiple negative sample pairs. This design can achieve parallel comparison of multiple negative samples in the same batch, significantly improving training efficiency. At the same time, by requiring the similarity of positive sample pairs to be higher than the similarity of all negative sample pairs, the model is forced to form more compact intra-class clustering and more dispersed inter-class separation in the semantic space, enhancing the discriminativeness of text representation. In addition, when the coverage of negative samples is sufficiently wide, sampling bias can be effectively avoided, enabling the model to learn more global and robust semantic features. In other embodiments of the present invention, a one-to-one contrastive form can also be used, such as constructing one positive sample pair and one negative sample pair for optimization. Although the efficiency of negative sample utilization is slightly lower than the former, it is applicable in small-batch training scenarios or specific tasks. Such variations should be considered as equivalent alternatives to the contrastive learning framework described in the present invention and fall within the protection scope of the present invention.

[0036] It should be noted that, through the aforementioned length bias suppression process, the original word sequence and the perturbed word sequence differ in length and local structure but remain consistent at the semantic level, constructing positive sample pairs. This ensures that the text vector model cannot rely solely on pseudo-features with similar lengths or similar positional distributions for similarity determination during training, effectively suppressing length bias and improving the stability of cross-length text comparison. On this basis, negative sample pairs are constructed by combining word sequences from the current training batch and other training batches, providing the model with "dissimilar" comparative references. By minimizing the similarity of negative sample pairs, the model is forced to capture more refined semantic features to distinguish subtle differences between different texts, especially in terms of sensitivity to temporal logic and structural differences. This embodiment, through the collaborative optimization of positive and negative sample pairs, enables the text vector model to be trained within a contrastive learning framework that aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. Ultimately, this forces the model to learn a high-quality text representation that is robust to length variations and enhances its ability to distinguish temporal and logical structural differences.

[0037] S4. Repeat steps S2 to S3 until the preset model convergence condition is met, and then output the trained text vector model.

[0038] The model convergence condition is any one or a combination of the following conditions: The model training iterations have reached the preset maximum number of iterations; The contrast loss value no longer decreases significantly or tends to stabilize over multiple consecutive training rounds; A complete traversal of all training batches in the original training text set has been completed; Once the model training reaches the preset convergence condition, the resulting text vector model can output more stable similarity results when faced with texts of varying lengths and changes in local word distribution, significantly improving its discrimination accuracy and generalization performance in the inference stage.

[0039] Example 2

[0040] This embodiment provides a text similarity comparison method, including: Obtain the first and second texts to be compared; The first and second texts to be compared are respectively input into the text vector model trained by the text vector model training method described above, and the first text representation vector and the second text representation vector are output. Based on the similarity function, the similarity score between the first text representation vector and the second text representation vector is calculated.

[0041] The similarity function can be the cosine similarity function: ; In the formula, The similarity score; This is the first text representation vector; This is the second text representation vector; Alternatively, the similarity function may also be dot product similarity or a calibrated learnable matching function; In some embodiments, the similarity score, as a quantitative indicator of text semantic similarity, can be directly applied to the following tasks: text pair similarity scoring, binary classification based on a preset threshold, Top-K result ranking in a retrieval system, sample association measurement in text clustering, or duplicate detection in large-scale text deduplication.

[0042] It should be noted that, since the text vector model is optimized during the training phase through length bias suppression enhancement and memory enhancement mechanisms, it learns robust semantic features with invariant length and has the ability to distinguish logical structure differences. Therefore, even if the input text has length differences or local redundant expressions during the application phase, the method in this embodiment can still output stable and accurate similarity results.

[0043] Example 3

[0044] like Figure 4 As shown, the present invention provides a text vector model training system 100, comprising: The training set acquisition module 101 is used to acquire the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches. The text processing module 102 is used to perform suppression length bias processing on the original word sequence of each training text in the current training batch to generate the corresponding perturbation word sequence. The loss calculation module 103 is used to construct positive and negative sample pairs for each training text based on the original word sequence and perturbed word sequence of each training text in the current training batch, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the pre-built text vector model. The model optimization module 104 is used to repeatedly execute the text processing module and the loss calculation module until the preset model convergence condition is met, and then output the trained text vector model.

[0045] Example 4

[0046] like Figure 5 As shown, the present invention provides a text similarity comparison system 200, comprising: The text acquisition module 201 is used to acquire the first and second texts to be compared. The vector generation module 202 is equipped with a text vector model trained using the text vector model training method described above. It is used to input the first text and the second text to be compared into the text vector model respectively, and output the first text representation vector and the second text representation vector. The text comparison module 203 is used to calculate the similarity score between the first text representation vector and the second text representation vector based on the similarity function.

[0047] Example 5

[0048] This embodiment provides an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] The non-transient software program and instructions required to implement the text vector model training method disclosed in Embodiment 1 or the text similarity comparison method disclosed in Embodiment 2 are stored in the memory. When executed by the processor, the method in Embodiment 1 or Embodiment 2 is executed. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0050] Example 6

[0051] This embodiment provides a computer-readable storage medium storing computer-executable instructions, which are used to cause a computer to execute the text vector model training method disclosed in Embodiment 1 above, or the text similarity comparison method disclosed in Embodiment 2.

[0052] It should be noted that the computer-readable storage medium in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0053] More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0054] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0055] The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device.

[0056] The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof. The aforementioned computer-readable storage medium can be used to write a computer program for performing this embodiment in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, and C++—and conventional procedural programming languages—such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0057] The above description is merely a preferred embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A text vector model training method, characterized in that, include: A. Obtain the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches; B. Perform a length-suppressed processing on the original word sequence of each training text in the current training batch to generate the corresponding perturbed word sequence; C. Based on the original word sequence and perturbed word sequence of each training text in the current training batch, construct positive sample pairs and negative sample pairs for each training text, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the pre-built text vector model. D. Repeat steps B to C until the preset model convergence condition is met, and then output the trained text vector model.

2. The method according to claim 1, characterized in that, The original word sequence of each training text is subjected to length-bias suppression processing, specifically including the following steps: With a preset repetition probability, at least one word position is randomly selected from the original word sequence of each training text to determine the set of words to be repeated. For each lexical in the set of lexicals to be repeated, insert a copy of that lexical at its adjacent position to generate a perturbed lexical sequence to suppress length bias. Determine whether the length of the perturbation word sequence exceeds the preset maximum sequence length: If the maximum sequence length is exceeded, the perturbation word sequence is truncated to meet the constraint of the maximum sequence length, and the truncated perturbation word sequence is output. If it does not exceed the limit, output the perturbation word sequence.

3. The method according to claim 1, characterized in that, Step C specifically includes the following steps: Based on the original word sequence and perturbed word sequence of each training text in the current training batch, construct positive sample pairs; Construct a negative sample set; The negative sample set includes the original word sequence and the perturbed word sequence corresponding to other training texts in the current training batch; Based on the original word sequence of each training text in the current training batch, construct negative sample pairs with the original word sequence or perturbed word sequence in the negative sample set; The original word sequence and the perturbation word sequence in the positive sample pair are respectively input into the pre-constructed text vector model to obtain the original text representation vector and the perturbation text representation vector, and the similarity of the positive sample pair is calculated based on the original text representation vector and the perturbation text representation vector. The original word sequence or perturbed word sequence in the negative sample pair is input into the text vector model to obtain the negative sample text representation vector, and the similarity of the negative sample pair is calculated based on the original text representation vector and the negative sample text representation vector. With the optimization objective of maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, a contrastive learning loss function is used to calculate the loss value, and the parameters of the text vector model are updated based on the loss value through a backpropagation algorithm.

4. The method according to claim 3, characterized in that, The original word sequence and the perturbation word sequence in the positive sample pair are input into a pre-constructed text vector model to obtain the original text representation vector and the perturbation text representation vector, respectively. Based on the original text representation vector and the perturbation text representation vector, the similarity of the positive sample pair is calculated, specifically including the following steps: The original word sequence and the perturbed word sequence in the positive sample pair are respectively input into the pre-built text vector model; The text vector model includes an encoder module, a memory enhancement module, a global semantic module, and a fusion module. The inputs of the memory enhancement module and the global semantic module are connected to the output of the encoder module, and the outputs of the memory enhancement module and the global semantic module are connected to the input of the fusion module. Based on the encoder module, the original word sequence and the perturbed word sequence are encoded respectively to obtain the corresponding original sequence features and perturbed sequence features; Based on the global semantic module, pooling operations are performed on the original sequence features and the perturbed sequence features respectively to obtain the corresponding first global semantic vector and second global semantic vector; According to the memory enhancement module, the original sequence features and the perturbation sequence features are sequentially updated and aggregated to obtain the corresponding first memory vector and second memory vector; According to the fusion module, the first global semantic vector and the first memory vector are fused to obtain the original text representation vector; According to the fusion module, the second global semantic vector and the second memory vector are fused to obtain the perturbation text representation vector.

5. The method according to claim 4, characterized in that, The memory enhancement module includes a memory update unit and a readout unit; The memory update unit is used to iteratively update the memory state of the current word according to the word order of the input sequence features, based on the features of the current word and the memory state of the previous word. The readout unit is used to generate output features based on the memory state corresponding to each word, and to perform pooling operations on the output features of all words to generate a memory vector.

6. The method according to claim 5, characterized in that, The memory enhancement module also includes a preprocessing unit; The preprocessing unit is used to perform local aggregation operations on the sequence features before the sequence features are input into the memory update unit.

7. The method according to claim 3, characterized in that, The negative sample set also includes the original lexical sequences and perturbed lexical sequences of the training texts from historical training batches.

8. A text similarity comparison method, characterized in that, include: Obtain the first and second texts to be compared; The first and second texts to be compared are respectively input into the text vector model trained by the text vector model training method as described in any one of claims 1 to 7, and the first text representation vector and the second text representation vector are output. Based on the similarity function, the similarity score between the first text representation vector and the second text representation vector is calculated.

9. A text vector model training system, characterized in that, include: The training set acquisition module is used to acquire the original training text set, perform normalization and word segmentation on each training text in the original training text set to obtain the corresponding original word sequence, and divide the original training text set into multiple training batches. The text processing module is used to perform suppression length bias processing on the original word sequence of each training text in the current training batch to generate the corresponding perturbation word sequence. The loss calculation module is used to construct positive and negative sample pairs for each training text based on the original and perturbed word sequences of each training text in the current training batch, and calculate the contrastive learning loss through the contrastive learning loss function to optimize the parameters of the pre-built text vector model. The model optimization module is used to repeatedly execute the text processing module and the loss calculation module until the preset model convergence condition is met, and then output the trained text vector model.

10. A text similarity comparison system, characterized in that, include: The text acquisition module is used to acquire the first and second texts to be compared. A vector generation module is deployed thereon with a text vector model trained using the text vector model training method as described in any one of claims 1 to 7, for inputting the first text and the second text to be compared into the text vector model respectively, and outputting the first text representation vector and the second text representation vector; The text comparison module is used to calculate the similarity score between the first text representation vector and the second text representation vector based on a similarity function.

Images