A method and device for matching enterprise unit addresses based on self-supervised representation learning

Abstract

The present invention discloses a method and device for matching corporate unit addresses based on self-supervised representation learning. The method first formulates positive and negative sample pairs according to specific application scenario rules, and at the same time, trains and learns a deep learning model through a combination of unsupervised and supervised methods. Through the combination of the two training methods, it is possible to learn more discriminative semantic representations in specific scenarios with a small number of labeled samples, and achieve accurate matching of corporate unit addresses. The method of the present invention is mainly divided into three parts: formulating rules for positive and negative training sample pairs, a model training method combining unsupervised and supervised methods, and matching unit addresses based on self-supervised representation learning. The method of the present invention can learn more discriminative semantic representations in specific scenarios with a small number of labeled samples, and achieve accurate matching of unit addresses, while outperforming the most advanced models in recent years in terms of evaluation indicators.

Description

Technical Field

[0001] The present invention relates to Chinese natural language processing fields such as financial enterprise unit address text sequences and e-commerce platform relevance matching, and more specifically, to a method and device for enterprise unit address matching based on self-supervised representation learning. Background Art

[0002] Text matching, or text relevance matching, is an important branch of natural language processing. By matching text relevance, we can discover users' potential interests and provide them with better content through relevant technical methods. It has important application value in finance, telecommunications, transportation, e-commerce, marketing and other fields. For example, on search platforms or in the e-commerce field, we often search for content of interest based on our needs. For example, topics such as "the most delicious Sichuan hot pot" or "Industrial and Commercial Bank of China in Xihu District, Hangzhou" are related. Because sometimes the search content and the content provided by the platform are not completely consistent, we need to make a relevance (similarity) judgment on the content displayed on the platform or related text to show users the most accurate content and improve the user stickiness of the platform and the company.

[0003] Currently, text matching processing technologies are mainly divided into two categories: Traditional machine learning methods: First, feature engineering is required, and the model can be established after feature engineering is completed. Its advantages are fast training speed and guaranteed accuracy, but the amount of feature engineering is huge; Deep learning methods: Deep learning models in this field are divided into two types: interaction-based models and representation-based models, such as Figure 1 As shown in Figure 2, the interaction model first fuses the two sequences for processing, obtains a feature vector through the model's hidden layer, and outputs the similarity at the output layer. The representation model extracts features from each of the two sequences, creates a vector representation, and finally calculates the similarity between the two vectors using a similarity algorithm.

[0004] The text matching techniques described above all have drawbacks. Traditional machine learning's feature engineering process is complex, requiring the creation of both text and character features, as well as interplay between features, and then evaluating their effectiveness. Deep learning approaches based on representation and interaction suffer from complex model structures, but are less effective in specific scenarios. For example, in the highly sequential context of corporate addresses, the text matching rules differ from those in general correlation scenarios. Swapping the order of address tokens can lead to irrelevance.

[0005] In summary, existing text matching technologies cannot simultaneously meet the following requirements:

[0006] 1) A general text processing framework;

[0007] 2) It has the ability to supplement text matching in specific scenarios;

[0008] 3) Evaluation indicators exceed the industry average. Summary of the Invention

[0009] In response to the problems caused by existing text matching processing methods, the present invention proposes a method and device for matching enterprise unit addresses based on self-supervised representation learning, and provides a text matching technology based on a deep learning model. By formulating positive and negative training samples and model training methods, it is possible to perform correlation or similarity matching between one or more pairs of texts. With a small number of labeled samples, it can learn more discriminative semantic representations in specific scenarios and achieve accurate matching of unit addresses. At the same time, it outperforms the most advanced models in recent years in terms of evaluation indicators.

[0010] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a method for matching enterprise unit addresses based on self-supervised representation learning, the method comprising the following steps:

[0011] (1) Obtain the address in the unit address matching scenario and construct the address data by marking it as an address token sequence;

[0012] (2) Initialize the positive sample set and the negative sample set. Initialize an integer K within the range of the address token length, traverse the address token sequence, randomly select K positions to modify the token, obtain positive and negative address token samples and add them to the corresponding sets. Concatenate all address strings and corresponding labels to obtain the positive sample training set and the negative sample training set.

[0013] (3) Construct a text matching model, which encodes the input text and obtains the encoded word vector representation. Then, an attention module is used to perform attention operations on the latent vector and word vector representing the last time step of the sentence vector to obtain the final weighted sentence vector.

[0014] (4) The text matching model is trained in a supervised learning manner based on the positive sample training set and the negative sample training set; then the text matching model is trained in an unsupervised manner, and the loss function of the self-supervised representation learning is optimized by back propagation to obtain the trained text matching model;

[0015] (5) Based on the trained text matching model, the text pairs to be matched are directly input into the text matching model in sequence to obtain the feature expression of the text. The similarity algorithm is used to calculate the similarity of the text feature vectors output by the text matching model. The calculated similarity is compared with the similarity threshold to obtain the enterprise unit address match.

[0016] Furthermore, in step (2), tokens are deleted, replaced, and abbreviated with a certain probability to generate positive samples, and then negative samples are generated by scrambling the tokens.

[0017] Furthermore, in step (4), the unsupervised learning method uses the Dropout mask in the model to perform two forward propagations on each sentence to obtain two different embeddings vectors. The vector pair obtained from the same sentence is used as a positive sample pair. For each vector, the embeddings vector generated by other sentences is selected as a negative sample to train the text matching model.

[0018] Furthermore, in step (4), the loss function of self-supervised representation learning is defined as follows:

[0019]

[0020] In the second aspect, the present invention provides an enterprise unit address matching device based on self-supervised representation learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the steps of the enterprise unit address matching method based on self-supervised representation learning.

[0021] In a third aspect, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the enterprise unit address matching method based on self-supervised representation learning.

[0022] Beneficial effects of the present invention:

[0023] 1. The enterprise unit address matching method proposed in this invention based on self-supervised representation learning can achieve accurate matching of unit addresses;

[0024] 2. The proposed method for matching unit addresses based on self-supervised representation learning and combining unsupervised and supervised approaches can learn more discriminative semantic representations for specific scenarios with a small number of labeled samples.

[0025] 3. The enterprise unit address matching device based on self-supervised representation learning and the computer-readable storage medium provided in the embodiments of the present invention can implement the steps of the enterprise unit address matching method based on self-supervised representation learning when executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a schematic diagram of deep text matching technology;

[0028] Figure 2 Schematic diagram of the self-supervised representation learning model structure;

[0029] Figure 3 Schematic diagram of the supervised learning process;

[0030] Figure 4 This is a structural diagram of an enterprise unit address matching device based on self-supervised representation learning according to the present invention. DETAILED DESCRIPTION

[0031] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0032] The present invention provides a method for matching enterprise unit addresses based on self-supervised representation learning. The specific steps are as follows:

[0033] (1) Obtain the address in the unit address matching scenario and construct the address data by marking it as an address token sequence;

[0034] (2) Initialize the positive sample set and the negative sample set. Initialize an integer K within the range of the address token length, traverse the address token sequence, randomly select K positions to modify the token, obtain positive and negative address token samples and add them to the corresponding sets. Concatenate all address strings and corresponding labels to obtain the positive sample training set and the negative sample training set.

[0035] (3) Construct a text matching model, which encodes the input text and obtains the encoded word vector representation. Then, an attention module is used to perform attention operations on the latent vector and word vector representing the last time step of the sentence vector to obtain the final weighted sentence vector.

[0036] (4) The text matching model is trained in a supervised learning manner based on the positive sample training set and the negative sample training set; then the text matching model is trained in an unsupervised manner, and the loss function of the self-supervised representation learning is optimized by back propagation to obtain the trained text matching model;

[0037] (5) Based on the trained text matching model, the text pairs to be matched are directly input into the text matching model in sequence to obtain the feature expression of the text. The similarity algorithm is used to calculate the similarity of the text feature vectors output by the text matching model. The calculated similarity is compared with the similarity threshold to obtain the enterprise unit address match.

[0038] The overall structure of the text matching model of the self-supervised representation learning of the present invention is as follows Figure 2 As shown, the input text is first encoded using a Bi-LSTM. After obtaining the word vector representations from the Bi-LSTM, an attention module is used to focus on the latent vector representing the last time step of the sentence vector and the word vector to obtain the final weighted sentence vector. Finally, the training method designed by the present invention is used to encode the positive and negative sample pairs separately, while a self-supervised representation learning loss function is set to optimize the representation learning model in the feature space.

[0039] First, we need to construct samples. In the scenario of unit address matching, the samples are all address sequences. We can construct the address data by labeling them as address token sequences. For example, label "Suzhou Industrial Park Loufeng School" as "Suzhou, R; Industry, I; Park, I; Loufeng, U; School, I;". The text matching model of the present invention adopts a combination of supervised and unsupervised methods to perform representation learning. The supervised learning method is to find the same or similar address sequence label samples as positive sample pairs (x, x + ), address tokens can be replaced with valid address sequences by statistically analyzing the training data or performing local data augmentation. These samples play a leading role in representation learning, guiding the encoder's representation range. The sample construction algorithm for supervised representation learning is as follows:

[0040]

[0041]

[0042] First initialize two sets: X + and X - , representing the positive and negative sample sets, respectively. An integer K is then initialized within the address token length range ADDRESS_LEN. As the address token sequence is traversed, tokens are randomly selected at K positions for modification. Specifically, tokens are deleted, replaced, and abbreviated with a certain probability to generate positive samples. Negative samples are then generated by shuffling them. Finally, the positive and negative address token samples are added to the corresponding sets. When the current address sequence is traversed, all address strings and corresponding labels are concatenated and output to the training set file.

[0043] Specific supervised learning methods, such as Figure 3 As shown in the figure. The main sample is fed into the encoder on the left, while the positive sample pairs are obtained by local augmentation through labeling on the right. The three samples on the right are enhanced using different methods, including removing repeated expressions, replacing and abbreviating some address tokens, and deleting tokens. This method simulates real-world user input expressions, allowing the model to narrow the representation distance between labels for a certain type of address sequence, improving the model's prediction accuracy. The dotted line at the bottom right represents a constructed negative example, ensuring that there are no repeated labels between negative sample pairs and that the corresponding sequence orders are different.

[0044] An unsupervised learning approach utilizes the dropout mask in the model to perform two forward passes on each sentence, generating two different embedding vectors. Pairs of vectors from the same sentence are used as positive examples, and for each vector, embedding vectors from other sentences are selected as negative examples to train the model. For models with dropout, dropout is enabled throughout the training process and varies across different samples during the forward pass. Therefore, no additional configuration is required; simply copy each sentence twice and feed it into the model. This unsupervised approach allows the model to learn an approximate representation of unit address sequences in this scenario, addressing data irregularities such as mistypes, reversals, and duplications often found in real-world text matching scenarios.

[0045] The model training process combines supervised and unsupervised methods. During the training process, the number of positive and negative sample pairs in each batch is different. In the supervised process, positive and negative sample pairs are directly constructed, so the ratio of positive and negative sample pairs can be randomly set in different batches to enhance the generalization of the model; in the unsupervised process, all samples need to be encoded twice through the encoder to obtain positive sample pairs. The dropout augmented embeddings generated by other different source sentences in the same batch are used as negative samples to form negative sample pairs. Therefore, in the unsupervised process, the ratio of positive and negative sample pairs is 1:N-1, where N is the total number of samples in the batch. Finally, the training of the model optimizes the loss function of self-supervised representation learning through backpropagation. The loss function of self-supervised representation learning is defined as follows:

[0046]

[0047] Among them, λ and 1-λ are balancing weights, the purpose of which is to increase the loss weight of the supervised representation learning sample part so that it plays a dominant role in representation learning. is the contrastive learning loss function for the supervised process, is the contrastive learning loss function for the unsupervised process.

[0048] Based on the trained model, text pairs to be matched can be directly input into the model to obtain the text's feature representation. Finally, a similarity algorithm can be used to calculate the similarity of the feature vectors. Depending on the application scenario, different probability intervals can be used to label the text pairs.

[0049] In the actual application of unit address matching, the method proposed in this invention is used to conduct an experimental comparison with the existing more advanced methods, as follows:

[0050] The embodiment of the present invention uses about 10,198 pieces of labeled data, and the specific data format is shown in Table 1:

[0051] Table 1

[0052]

[0053]

[0054] Here, 1 represents a perfect match, and 0 represents a mismatch. Then, there are approximately 900,000 unlabeled data items. This unlabeled data item undergoes the aforementioned word segmentation and construction process. This also generates some labeled data, bringing the total number of labeled data items to 90,000. This represents 900,000 unlabeled data items. The method of the present invention is used to train a self-supervised representation learning model.

[0055] The embodiment of the present invention is implemented based on the Tensorflow deep learning framework. The character word vectors in the model structure are initialized using the pre-trained ELMo word vector, the dimension of the character word vector is 200, and the size of the dictionary is 5k. These character word vectors will be updated during the training process. For λ in the loss function, it is empirically set to 0.7. In order to achieve unsupervised representation learning, dropout is used before the Bi-LSTM encoder and the sentence latent vector are paid attention, and the probability of dropout is set to 0.5 in the present invention. The hidden state dimensions of the Bi-LSTM are respectively set to 200 in the present invention. The present invention uses an RTX-Titan graphics processor (GPU) for training.

[0056] In addition, the present invention conducted two sets of control experiments. The first set of experiments compared the proposed method with SimCSE, a recently proposed contrastive learning paradigm, which uses an unsupervised approach to learn representations on an address dataset. The second set of experiments compared the method with Sentence-Bert, a classic text matching model. The present invention also conducted an indicator analysis on the manual address pair dataset.

[0057] The specific experimental results are shown in Table 2:

[0058] Table 2

[0059]

[0060] As can be seen, the model of the present invention achieves the best results in terms of AUC and F1. On the artificially created dataset, the Pair-MSE index of the present invention can reach the lowest, demonstrating the effectiveness and generalizability of the model in the unit address text matching scenario.

[0061] Corresponding to the aforementioned embodiment of the enterprise unit address matching method based on self-supervised representation learning, the present invention also provides an embodiment of the enterprise unit address matching device based on self-supervised representation learning.

[0062] See also Figure 4 An embodiment of the present invention provides an enterprise unit address matching device based on self-supervised representation learning, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the enterprise unit address matching method based on self-supervised representation learning in the above embodiment.

[0063] The embodiment of the enterprise unit address matching device based on self-supervised representation learning of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown, this is a hardware structure diagram of any device with data processing capability where the enterprise unit address matching device based on self-supervised representation learning of the present invention is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0064] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0065] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0066] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the enterprise unit address matching method based on self-supervised representation learning in the above embodiment is implemented.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0068] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for matching enterprise unit addresses based on self-supervised representation learning, characterized in that: The method comprises the following steps: (1) Obtain the address in the unit address matching scenario and construct it by marking the address data into an address token sequence; (2) Initialize the positive sample set and the negative sample set, and initialize an integer within the range of the address token length , traverse the address token sequence and randomly select Modify the token at each position to obtain positive and negative address token samples and add them to the corresponding sets. Concatenate all address strings and corresponding labels to obtain positive sample training sets and negative sample training sets. (3) Construct a text matching model, which encodes the input text, obtains the encoded word vector representation, and then uses the attention module to perform attention operations on the latent vector and word vector representing the last time step of the sentence vector to obtain the final weighted sentence vector; (4) The text matching model is trained in a supervised learning manner based on the positive sample training set and the negative sample training set; then the text matching model is trained in an unsupervised manner, and the loss function of the self-supervised representation learning is optimized by back propagation to obtain the trained text matching model; The unsupervised learning method uses the Dropout mask in the model to perform two forward propagations on each sentence to obtain two different embeddings vectors. The vector pairs obtained from the same sentence are used as positive sample pairs. For each vector, the embedding vectors generated by other sentences are selected as negative samples to train the text matching model. The loss function L for self-supervised representation learning is defined as follows: in and It is a balancing weight, the purpose of which is to increase the loss weight of the supervised representation learning sample part so that it plays a leading role in representation learning; is the contrastive learning loss function for the supervised process, is the contrastive learning loss function for the unsupervised process; (5) Based on the trained text matching model, the text pairs to be matched are directly input into the text matching model in sequence to obtain the feature expression of the text. The similarity algorithm is used to calculate the similarity of the text feature vectors output by the text matching model. The calculated similarity is compared with the similarity threshold to obtain the enterprise unit address match.

2. The enterprise unit address matching method based on self-supervised representation learning according to claim 1 is characterized in that: In step (2), tokens are deleted, replaced, and abbreviated with a certain probability to generate positive samples, and then negative samples are generated by scrambling the tokens.

3. A device for matching enterprise unit addresses based on self-supervised representation learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the steps of the enterprise unit address matching method based on self-supervised representation learning as described in any one of claims 1-2.

4. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the enterprise unit address matching method based on self-supervised representation learning as described in any one of claims 1 to 2 are implemented.

Images