A Dense Vector Retrieval Method for Structured Data

Abstract

This invention belongs to the field of structured data retrieval technology and discloses a dense vector retrieval method for structured data. The method involves: establishing a language model structure; collecting structured and unstructured data pairs to identify entities in the structured data; aligning the structured and unstructured data; optimizing the vector space in the language model through comparative learning of existing structured and unstructured data; entity mask prediction; and using mask language modeling to learn the latent structural information in the structured data from the identified entities, further optimizing the vector space in the language model. The proposed method achieves state-of-the-art results in code search and product search. Furthermore, our model exhibits strong zero-shot capability, even surpassing the performance of fine-tuned models on some datasets.

Description

Technical Field

[0001] This invention relates to the field of structured data retrieval technology, and in particular to a dense vector retrieval method for structured data. Background Technology

[0002] Dense retrieval has demonstrated powerful performance in many natural language processing applications, such as open-domain question answering, dialogue search, and fact verification. Given a user query and data documents, a dense retrieval engine uses a language model to encode the query and data documents, maps them to a vector space for matching, and returns data documents that meet the user's query requirements. However, many real-world structured data retrieval scenarios exist, such as code retrieval and product retrieval tasks. These structured data retrieval tasks require language models to retrieve corresponding structured data for user queries in a dense retrieval manner to obtain the structured data items that the user needs. Therefore, learning representations of structured data is crucial for building more comprehensive retrieval systems.

[0003] —Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP In 2020, pages 1536–1547, CodeBERT was the first large-scale pre-trained natural language / programming language model. This pre-trained model could handle many natural language / programming language problems, such as searching for code in natural language and generating code. The model training design mainly included two training objectives: masked language modeling and replaceable word detection. Objective 1: Masked Language Modeling. Natural language / programming language pairs were used as input, and characters were randomly selected from both languages ​​and replaced using special masked characters. Objective 2: Replaceable Word Detection, similar to ELECTRA. However, CodeBERT's shortcoming lies in its near-universal adoption of pre-training strategies for unstructured data, namely traditional masked language modeling and ELECTRA-based adversarial learning. It did not propose corresponding pre-training strategies for the structural information inherent in structured data.

[0004] —Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages In sections 1536–1547, CodeT5 is a unified pre-trained encoder-decoder model, sharing the same overall architecture as T5. It better leverages the code semantics conveyed in identifiers assigned by developers. The model employs a unified framework to support code understanding and generation tasks, and models various code learning tasks. It also proposes a novel identifier-aware pre-training task, enabling the model to distinguish which code characters are identifiers. Furthermore, it utilizes user-written code comments and a bimodal bi-generation task for better natural language-programming language alignment. However, CodeT5's weakness lies in its failure to effectively learn vector representations of structured data. While CodeT5 mimics previous unsupervised training methods, designing specific masking strategies and unique generative alignment tasks for specific structured data, it lacks effective structured data representation learning, limiting its performance on structured data retrieval tasks.

[0005] In the paper "Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. 2022. CodeRetriever: Large-scale Contrastive Pre-training for Code Search," CodeRetriever proposes unimodal and bimodal contrastive learning strategies for code retrieval tasks, with the same model architecture and initial parameters as GraphCodeBERT. For unimodal contrastive learning, it constructs code pairs with similar functions in an unsupervised manner based on natural language information in the code. For bimodal contrastive learning, it constructs code-text pairs using the code's documentation and scattered comments within the code. However, CodeRetriever's limitation lies in its contrastive learning only for cross-modal and same-modal data, initially optimizing the vector representation of structured data, without further learning the structural information of structured data, resulting in insufficient structure awareness. Summary of the Invention

[0006] The technical problem this invention aims to solve is: how to design a novel dense vector retrieval model for structured data, effectively understanding and learning the latent structural knowledge within structured data to improve the retrieval performance of structured data. Based on this technical problem, the purpose of this invention is to propose a model that can utilize the potential capabilities of current pre-trained language models, further training them to acquire structure-awareness capabilities, making them more suitable for structured data retrieval tasks. To bridge the modal gap between structured and unstructured data and better represent structured data, a structured data alignment task is used to train the language model, aligning and matching structured and unstructured data in the vector space. Simultaneously, to further capture semantic information from structured data, a masked entity prediction task is employed to mask entities in structured data, training the language model to predict the masked portions of the structured data.

[0007] The technical solution of this invention: a dense vector retrieval method for structured data, specifically including the following steps:

[0008] Step 1: Establish the language model structure, based on the T5 model; the T5 model is based on the standard Transformer structure;

[0009] Step 2: Data processing; collect structured and unstructured data pairs, and identify entities in the structured data;

[0010] Step 3: Align structured and unstructured data; train the language model by comparing and contrasting existing structured and unstructured data, and optimize the vector space in the language model;

[0011] Step 4: Entity masking prediction; The entities in the structured data identified in Step 2 are trained using masking language modeling to further optimize the vector space in the language model;

[0012] Step 2 specifically involves:

[0013] Step 2.1: Data collection. Collect pre-training data from existing open-source datasets, and collect corresponding structured and unstructured data pairs for the two retrieval scenarios of code retrieval and product retrieval.

[0014] Step 2.2: Data processing. For the collected pre-training data, parse the structured data and extract the entities from the structured data.

[0015] For code retrieval scenarios, code data is collected, including structured code language data and corresponding unstructured document descriptions; for product retrieval scenarios, product data is collected, including structured product description data and unstructured product key information data.

[0016] In the structured code language data, identifiers in the code are entities, which are identified and obtained using the tree_sitter tool; the identifiers include code variables, function names, external libraries, and methods; in the structured product description data, nouns and special nouns that appear in both the product description and the product title are entities, which are identified and obtained using the nltk tool; the nouns and special nouns are used to describe the attributes of the product.

[0017] Step 3 specifically involves:

[0018] Step 3.1: Based on the data construction in Step 2, for each piece of structured data, there is corresponding unstructured data with the same semantics; the language model selected in Step 1 encodes the unstructured and structured data into vectors respectively; for unstructured data, it and its corresponding structured data are regarded as positive sample pairs, and it and the remaining structured data are regarded as negative sample pairs; through comparative learning using the cross-entropy loss function, positive sample pairs are brought closer together and negative sample pairs are separated in the vector space of the language model;

[0019] Step 3.2: Backpropagate the loss value obtained from the above comparative learning to update the parameters of the language model, align structured and unstructured data, and optimize the vector space of the language model;

[0020] Step 4 specifically involves:

[0021] Step 4.1: In the entity prediction task, the language model is guided to better understand the semantics of structured data by recovering the masked entities. For the entities identified in Step 2, they are replaced with special labels from the T5 model to obtain the masked structured data, and these entities are combined into an entity sequence; for a structured data set: {X1, X2, ENT1, X3, ENT2, ..., X...} n The structured data of entities appearing in a structured data document is obtained by using the special marker mask:

[0022] X3,<extra_id_1> , ..., X n}, from steps

[0023] The obtained entities are then combined into an entity sequence in the following form: Y d ={<extra_id_0> ENT1, ...,<extra_id_n> ENT n},in<extra_id_i> Let X represent the (i+1)th masked entity, X be other words in the structured data, and ENT be the entity identified in step 2.2. The structured data of the masked entity and the corresponding entity sequence are obtained through the above method. We use the above masked structured data as the input to the encoder of the language model and the entity sequence as the prediction result of the decoder of the language model, and then obtain the entity predicted by the language model.

[0024] Step 4.2: The encoder of the language model encodes the structured data of the input masked entities into a context vector, and inputs it into the decoder to predict and generate the masked entities one by one in an autoregressive manner, thereby obtaining a sequence of predicted entities; the loss value between the predicted entities and the real entities is obtained through the cross-entropy loss function, and the language model is further optimized to better capture entity information and understand the semantics of structured data.

[0025] The special tags are derived from the vocabulary of the T5 model, and are in the form of {<extra_id_0> ,<extra_id_1> ,...,<extra_id_99>}

[0026] The beneficial effects of this invention: The pre-training method proposed in this invention achieves state-of-the-art results in code retrieval and product retrieval. Experimental results show that our model achieves excellent results on multiple code retrieval datasets, including six programming languages: Ruby, Javascript, Go, Python, Java, and PHP, surpassing several pre-trained language models, including CodeRetriever. It also achieves excellent results on product retrieval datasets, outperforming existing pre-trained language models. Furthermore, our model exhibits strong zero-shot capability, even surpassing the performance of fine-tuned versions of other models on some datasets. Attached Figure Description

[0027] Figure 1 A flowchart of a dense vector retrieval method for structured data;

[0028] Figure 2 This is a schematic diagram of structured data alignment and masked entity prediction. Detailed Implementation

[0029] The specific process of a dense vector retrieval method for structured data is as follows:

[0030] Step 1: Establish the language model structure;

[0031] Step 2: Data processing; collect structured and unstructured data pairs, and identify entities in the structured data;

[0032] Step 3: Align structured and unstructured data; use existing structured and unstructured data for comparative learning to optimize the vector space in the language model;

[0033] Step 4: Entity mask prediction; The entities in the identified structured data are modeled using a mask language to learn the potential structural information in the structured data, and the vector space in the language model is further optimized.

[0034] Furthermore, the steps are specifically implemented by performing the following sub-steps:

[0035] Step 2.1: First, we collect the data we need from existing datasets. For the code retrieval scenario, we collect pre-training data from the CodesearchNet dataset, which includes six code languages: Ruby, Javascript, Go, Python, Java, and PHP. These structured codes all have corresponding documentation and functional descriptions. For the product retrieval scenario, we collect pre-training data from the ESCI dataset. Each product has both structured product description information and unstructured product highlights. Using this collected data, we construct the required pre-training dataset.

[0036] Step 2.2: Use existing tools to parse structured data and identify entities within it. For code, we treat identifiers such as variables, function names, external libraries, and methods as entities, and retain comments and more code information for commented code, as comments are typically only found in longer and more complex code, and retaining this information helps with alignment. We use the tree_sitter tool to identify these entities in the code. For product data, we use the nltk tool to identify entities in product descriptions, which are nouns and special nouns that appear in both product descriptions and product titles.

[0037] Step 3.1: Based on the data construction in Step 2, for each structured data, there is unstructured data with the same semantics; input the structured data and unstructured data into the encoder of the language model respectively, and obtain the structured data d and unstructured data p encoded as vectors at the decoder.

[0038] Step 3.2: After obtaining the vector encodings of unstructured and structured data, contrastive learning is used to continue training the language model; for unstructured data p, we treat the matched structured data as positive samples d. + The remaining structured data is considered as negative samples d - In vector space, p and d + Zoom in and compare p with d - Separation; this contrastive training method allows the model to learn better representations of structured data.

[0039] Step 4.1: In the entity prediction task, for each entity obtained in Step 2, replace them with the same special label; for a structured data set: {X1, X2, ENT1, X3, ENT2, ..., X...} n The structured data of the masked entity is obtained by applying the special marker mask to the entity appearing in the structured data. X3,<extra_id_1> , ..., X nThe entities obtained in step 2 are then combined into an entity sequence in the following form: Y d ={<extra_id_0> ENT1, ...,<extra_id_n> ENT n},in<extra_id_i> Let X represent the (i+1)th masked entity, X be other words in the structured data, and ENT be the entity identified in step 2.2. The structured data of the masked entity and the corresponding entity sequence are obtained in the above manner. The structured data of the masked entity is used as the input to the language model encoder, and the entity sequence is used as the input to the language model decoder.

[0040] Step 4.2: The encoder of the language model encodes the structured data of the input masked entity into a context vector, and inputs it into the decoder to predict the masked entity one by one through autoregression, so as to obtain the predicted entity sequence; the loss value between the predicted entity sequence and the entity sequence is calculated by the cross-entropy loss function, and the language model parameters are updated by backpropagation.

[0041] This invention proposes two pre-training methods to enable language models to have structure awareness and learn effective structured data representations: 1) Structured data alignment, utilizing the natural alignment relationships between structured and unstructured data for structure-aware pre-training; 2) Masked entity prediction, designing an entity-oriented masking strategy and requiring the language model to predict the masked entities.

[0042] Table 1 compares the performance of this invention with existing models applied to code retrieval and product retrieval scenarios. MRR stands for Mean Reciprocal Ranking, and NDCG is the Normalized Diminished Cumulative Gain. Both are metrics used to measure the model's retrieval ranking performance. Table 1 shows that this invention achieves the best results in both code retrieval and product retrieval scenarios, particularly demonstrating strong zero-shot capability. In the code retrieval scenario, the retrieval performance of this invention not only surpasses pre-trained language models such as BERT and T5, as well as code-pre-trained language models specifically trained on code data like CodeBERT and CodeT5, but also surpasses the CodeRetriever model specifically designed for code retrieval. In the product retrieval scenario, the performance of this invention also surpasses that of previous pre-trained language models.

[0043] Table 1 Comparison of Model Performance

[0044]

[0045]

Claims

1. A dense vector retrieval method for structured data, characterized in that, The specific steps are as follows: Step 1: Establish the language model structure, based on the T5 model; the T5 model is based on the standard Transformer structure; Step 2: Data processing; Collect pairs of structured and unstructured data, and identify entities within the structured data; Step 3: Align structured and unstructured data; optimize the vector space of the language model by comparing and learning from existing structured and unstructured data; Step 3 specifically involves: Step 3.1: Based on the data construction in Step 2, for each piece of structured data, there is corresponding unstructured data with the same semantics; the language model selected in Step 1 encodes the unstructured and structured data into vectors respectively; for unstructured data, it and its corresponding structured data are regarded as positive sample pairs, and it and the remaining structured data are regarded as negative sample pairs; the language model is trained by comparison learning through the cross-entropy loss function, which brings positive sample pairs closer and separates negative sample pairs in the vector space of the language model; Step 3.2: Backpropagate the loss value obtained through the above comparative learning to update the parameters of the language model, align structured and unstructured data, and optimize the vector space of the language model; Step 4: Entity mask prediction; The entities in the structured data identified in Step 2 are trained using mask language modeling to further optimize the vector space of the language model; The search scenarios include code search scenarios and product search scenarios; for code search scenarios, code data is collected, including structured code language data and corresponding unstructured document descriptions; For product retrieval scenarios, collect product data, including structured product description data and unstructured product key data; In the structured code language data, identifiers in the code are entities; identifiers in the code are identified and obtained using the tree_sitter tool; the identifiers include variables, function names, external libraries, and methods in the code; in the structured product description data, nouns and special nouns that appear in both the product description and the product title are entities; nouns and special nouns are identified and obtained using the nltk tool; the nouns and special nouns are used to describe the attributes of the product.

2. The dense vector retrieval method for structured data according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Data collection. Collect pre-training data from existing open-source datasets and collect structured and unstructured data pairs for the corresponding retrieval scenarios; Step 2.2: Data processing. For the collected pre-training data, parse the structured data and obtain the entities in the structured data.

3. The dense vector retrieval method for structured data according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1: In the entity prediction task, for each entity obtained in Step 2, replace them with the same special label; for a structured data set: {X1, X2, ENT1, X3, ENT2, ..., X...} n The structured data of entities appearing in a structured data document is obtained by using the special marker mask: The entities obtained from step 2 are then combined into an entity sequence in the following form: Y d ={<extra_id_0> ENT1, ...,<extra_id_n> ENT n },in<extra_id_i> Let X represent the (i+1)th masked entity, X be other words in the structured data, and ENT be the entity identified in step 2.

2. The structured data of the masked entity and the corresponding entity sequence are obtained in the above manner. The structured data of the masked entity is used as the input to the language model encoder, and the entity sequence is used as the prediction result of the language model decoder. Step 4.2: The encoder of the language model encodes the structured data of the input masked entities into a context vector, and inputs it into the decoder to predict and generate the masked entities one by one in an autoregressive manner, thereby obtaining the predicted entity sequence; the loss value between the predicted entity sequence and the entity sequence is calculated by the cross-entropy loss function, and the language model parameters are updated by backpropagation.

4. The dense vector retrieval method for structured data according to claim 3, characterized in that, The special tags are derived from the vocabulary of the T5 model, and are in the form of {<extra_id_0> ,<extra_id_1> ,...,<extra_id_99>}

Images