API document recommendation method based on large language model

Abstract

The invention discloses an API document recommendation method based on a large language model. The method comprises the steps of obtaining a plurality of text queries and a plurality of API documents, and presetting correlation values, so as to construct a reordering data set; training the constructed reordering model according to the reordering data set to obtain a trained reordering model; obtaining a to-be-tested text query, preprocessing all the API documents, and obtaining an initial sorting API document recommendation sequence of the to-be-tested text query according to the to-be-tested text query and the preprocessed API documents; and processing the initially sorted API document recommendation sequence and the to-be-tested text query by adopting the trained resorting model to obtain a resorted API document recommendation sequence of the to-be-tested text query. According to the method, the API document related to the development task is automatically recommended by understanding the query intention of developers, so that the consulting time of the developers is effectively shortened, and the development efficiency is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of API document recommendation, and particularly relates to an API document recommendation method based on a large language model. BACKGROUND

[0002] With the continuous development and increasing complexity of software development, the challenges faced by developers are also increasing. Especially when building and maintaining large software systems, developers often need to consult a large number of application programming interface (API) documents to understand and correctly use different libraries, frameworks and tools. However, as the number of APIs used by developers increases, the workload of manually searching for relevant documents also increases significantly, which not only wastes a lot of time and effort, but also easily causes a decrease in development efficiency. Therefore, how to quickly and accurately recommend relevant API documents has become an important problem for improving development efficiency and optimizing development process.

[0003] Early API document recommendation systems mainly rely on traditional rule-based or keyword matching techniques. With the advancement of deep learning technology, the introduction of embedding vector models has significantly improved the performance of natural language understanding tasks. These models construct a vector space by learning a large amount of programming documents and code data, and generate recommendation results with certain semantic understanding through vector similarity comparison. Although these methods can provide basic document recommendations in some scenarios, they often fail to fully consider the development context and specific needs of developers. Especially in the face of complex and dynamic development environments, existing systems are difficult to simultaneously meet the intentions of developers and the context of code, thereby affecting the accuracy and practicality of the recommendation results. SUMMARY

[0004] In order to solve the problems in the background art, the application provides an API document recommendation method based on a large language model, which solves the technical problem that the prior art cannot fully consider the development context and specific needs of developers.

[0005] The technical solution adopted by the application includes:

[0006] One, an API document recommendation method based on a large language model, comprising the following steps:

[0007] S1, obtaining a plurality of text queries and a plurality of API documents, each text query and each API document being concatenated to obtain text input data, each text input data being pre-set with a relevance value, the text input data and the corresponding relevance value being combined into reordering data, and all reordering data being summarized to obtain a reordering data set.

[0008] S2, construct a reordering model, input the text input data of the reordered data as input, and input the corresponding relevance value as label, input the reordered data set into the reordering model for training, and obtain the trained reordering model.

[0009] S3, obtain a to-be-tested text query, preprocess all API documents obtained in step S1, and obtain an initial ordering API document recommendation sequence of the to-be-tested text query according to the to-be-tested text query and the preprocessed API documents.

[0010] S4, using the trained reordering model, processing the initial ordering API document recommendation sequence and the to-be-tested text query to obtain a reordering API document recommendation sequence of the to-be-tested text query.

[0011] After obtaining the reordering API document recommendation sequence, the developer can search according to each API document in the reordering API document recommendation sequence.

[0012] The step S1 specifically comprises: S11, obtaining a plurality of text problems of a developer and a code segment corresponding to each text problem, and splicing each text problem and the corresponding code segment to obtain a text query of the developer.

[0013] S12, collecting a plurality of API documents, splicing each text query and each API document to obtain text input data, predefining a relevance value between the corresponding text query and API document of each text input data, and combining the text input data and the corresponding relevance value as reordering data, and all reordering data are summarized to obtain a reordering data set.

[0014] The reordering model comprises a natural language extractor, an inference state extractor and a relevance detector connected in sequence; the natural language extractor is a pre-trained natural language model and a full connection layer connected in sequence; the inference state extractor is a plurality of state extraction layers connected in sequence; the reordering data set is output to the input end of the pre-trained natural language model for processing, the result output by the natural language model is input to the full connection layer for processing, the result output by the full connection layer is input to the plurality of state extraction layers connected in sequence for processing, and the result output by the last state extraction layer is input to the relevance detector for processing, and the result output by the relevance detector is taken as the output of the reordering model.

[0015] Each state extraction layer is set according to the following formula:

[0016] s=LayerNorm(FFN(p″)+p″)

[0017] p″=CrossAttention(p′,h′,h′)

[0018] p' = SelfAttention(p, p, p)

[0019] where s represents the output of the state extraction layer; LayerNorm() represents a normalization layer; FFN() represents a feedforward network layer; CrossAttention() represents a cross-attention mechanism layer; SelfAttention() represents a self-attention mechanism layer; p represents a learning vector; h' represents the output of the previous layer of the current state extraction layer; p' represents the output of the self-attention mechanism; and p" represents the output of the cross-attention mechanism layer.

[0020] The correlation detector is set according to the following formula:

[0021] r = W * g" + b

[0022] g" = LayerNorm(g' + FFN(g'))

[0023] g' = LayerNorm(MHA(g, s, s))

[0024] where r represents the output of the correlation detector, and also represents a correlation value; W represents a learning weight; b represents a bias; LayerNorm() represents a normalization layer; FFN() represents a feedforward network layer; MHA() represents a multi-head attention mechanism layer; g represents a learning vector; s represents the output of the last state extraction layer in the inference state extractor; g' represents the output of the first normalization layer in the correlation detector; and g" represents the output of the second normalization layer in the correlation detector.

[0025] The loss function of the reordering model is set according to the following formula:

[0026]

[0027] where Loss represents the loss function of the reordering model; K represents the number of reordering data in a training batch; represents an expectation; q represents a text question in the reordering data; i and j both represent indexes; d i and d j respectively represent the API document of the i-th reordering data and the API document in the j-th reordering data; D represents a reordering data set; σ represents a logistic function; r represents a correlation value; and the API document d i of the i-th reordering data corresponds to a correlation value greater than the API document d j of the j-th reordering data; (q, d i , d j ) represents a triple randomly sampled from the reordering data set D. denotes the expectation of all triples sampled from the reordering dataset D.

[0028] The step S3 is specifically: S31, obtaining a text question to be tested and a corresponding code segment, and splicing the text question to be tested and the corresponding code segment to obtain a text query to be tested.

[0029] S32, each API document obtained in the step S1 is mapped into a document vector feature by using a vector retrieval model, and each API document and the corresponding document vector feature are stored into a constructed vector retrieval library.

[0030] S33, the text query to be tested is mapped into a text vector feature to be tested by using the vector retrieval model; the cosine similarity value between the text vector feature to be tested and each document vector feature in the vector retrieval library is calculated, the first N API documents are selected from the vector retrieval library according to the cosine similarity value from large to small, and a primary sorting API document recommendation sequence of the text query to be tested is obtained.

[0031] The vector retrieval model uses a UnixCoder model.

[0032] The step S4 is specifically: S41, splicing the text query to be tested and each API document in the primary sorting API document recommendation sequence to obtain N text input data to be tested.

[0033] S42, the text input data to be tested is input into the trained reordering model for processing, and the relevance value of each text input data to be tested is obtained.

[0034] S43, the first M API documents are selected from the primary sorting API document recommendation sequence according to the relevance value from large to small, and a reordering API document recommendation sequence of the text query to be tested is obtained.

[0035] II. A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] The beneficial effects of the present application are:

[0037] The present application trains a reordering model with the ability to understand and predict the relevance of API documents and text queries by using the relevance data of API documents and text queries of developers, reorders the primary sorting document retrieval results, and thus provides the API documents that better match the intention and development scene of the developers for the developers, and improves the development efficiency and optimizes the development process. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the reasoning phase of the method of the present application.

[0039] Figure 2 Flow chart for training a reordering model for the method of the present invention.

[0040] Figure 3 Flow chart for constructing a vector retrieval library for the method of the present invention.

[0041] Figure 4 Module chart for a reordering model in the method of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in more detail below with reference to the drawings and embodiments, but the present invention is not limited thereto. Those skilled in the art can make several improvements and refinements without departing from the principles of the present invention, and these improvements and refinements are also considered to be within the scope of the present invention. The contents not described in detail in the present specification are conventional technologies known to those skilled in the art.

[0043] The API document recommendation method of the present embodiment includes the following steps:

[0044] S1, obtain a plurality of text queries and a plurality of API documents, splice each text query and each API document to obtain text input data, predefine a relevance value for each text input data, combine the text input data and the corresponding relevance value as reordering data, and aggregate all the reordering data to obtain a reordering data set.

[0045] S11, obtain a plurality of text problems (user's intention) of a developer and a code snippet corresponding to each text problem, splice each text problem and the corresponding code snippet to obtain a text query of the developer.

[0046] S12, collect a plurality of API documents, splice each text query and each API document to obtain text input data, predefine a relevance value between the corresponding text query and API document for each text input data, combine the text input data and the corresponding relevance value as reordering data, and aggregate all the reordering data to obtain a reordering data set.

[0047] Each API document includes function description, parameter description, return value type, usage example, etc.

[0048] In the present embodiment, as Figure 1The "troch.dot" API document shown in the "vector retrieval library construction process" is a specific API document, including the function description "torch.dot(input, other) calculates the dot product between two one-dimensional tensors (vectors)", the parameter description "input (Tensor): a one-dimensional tensor, other (Tensor): another one-dimensional tensor, which must have the same size as input", the return value type "torch.Tensor, returns a scalar tensor representing the dot product of the two input tensors.", and a usage example as shown in Figure 1

[0049] As shown in Figure 2 S2, a reordering model is constructed in the computer, taking the text input data of the reordered data as input and the corresponding relevance value as label, inputting the reordered data set into the reordering model for training to obtain the trained reordering model.

[0050] The reordering model includes a natural language extractor, an inference state extractor and a relevance detector connected in sequence; the natural language extractor is a pre-trained natural language model (LLM layer) and a fully connected layer connected in sequence; the inference state extractor is a plurality of state extraction layers connected in sequence; the reordered data set is output to the input end of the pre-trained natural language model for processing, the result output by the natural language model is input to the fully connected layer for processing, and the result output by the fully connected layer is input to the plurality of state extraction layers connected in sequence for processing, and the result output by the last state extraction layer is input to the relevance detector for processing. The result output by the relevance detector is taken as the output of the reordering model.

[0051] Each state extraction layer includes a self-attention mechanism layer (SelfAttention), a cross-attention mechanism layer (CrossAttention), a feedforward network layer (FFN) and a layer normalization; the relevance detector includes a multi-head self-attention mechanism layer (MHA), a normalization (LayerNorm), a feedforward network layer (FFN) and a fully connected layer.

[0052] The fully connected layer is set according to the following formula:

[0053] h1=W1*h+b1

[0054] Where h1 represents the output of the fully connected layer; h represents the output of the pre-trained natural language model; W1 represents the learning weight, and b1 represents the bias.

[0055] In this embodiment, the pre-trained natural language model adopts CodeLlama model.

[0056] In specific implementation, as Figure 4 ​As shown in the snowflake, the CodeLlama model is directly adopted after pre-training is completed, and all parameters are frozen. During training of the reordering model, all parameters of the CodeLlama model are frozen and not updated. As shown in the flame, the parameters of the inference state extractor and the correlation detector are updated during the training process. Figure 4

[0057] Each state extraction layer is set according to the following formula:

[0058] s = LayerNorm(FFN(p") + p")

[0059] p" = CrossAttention(p', h', h')

[0060] p' = SelfAttention(p, p, p)

[0061] where s represents the output of the state extraction layer; LayerNorm() represents a normalization layer; FFN() represents a feedforward network layer; CrossAttention() represents a cross-attention mechanism layer; SelfAttention() represents a self-attention mechanism layer; p represents a learning vector; h' represents the output of the previous layer of the current state extraction layer; p' represents the output of the self-attention mechanism; and p" represents the output of the cross-attention mechanism layer.

[0062] In specific implementation, when the state extraction layer is the first state extraction layer of the reordering model, h' represents the output h1 of the fully connected layer; and when the state extraction layer is any subsequent state extraction layer in the reordering model except the first state extraction layer, h' represents the output of the previous state extraction layer of the current state extraction layer.

[0063] The correlation detector is set according to the following formula:

[0064] r = W * g" + b

[0065] g" = LayerNorm(g' + FFN(g'))

[0066] g' = LayerNorm(MHA(g, s, s))

[0067] where r represents the output of the correlation detector, and also represents a correlation value; W represents a learning weight; b represents a bias; LayerNorm() represents a normalization layer; FFN() represents a feedforward network layer; MHA() represents a multi-head attention mechanism layer; g represents a learning vector; s represents the output of the last state extraction layer in the inference state extractor; g' represents the output of the first normalization layer in the correlation detector; and g" represents the output of the second normalization layer in the correlation detector. ​

[0068] The loss function of the reordering model is set according to the following formula:

[0069]

[0070] wherein Loss represents the loss function of the reordering model; K represents the number of reordering data in a training batch; represents expectation; q represents a text question in the reordering data; i and j both represent indexes; d i and d j respectively represent an API document of the i-th reordering data and an API document in the j-th reordering data; D represents a reordering data set; σ represents a logistic function; r represents a correlation value; and the API document d i of the i-th reordering data corresponds to a correlation value greater than the API document d j of the j-th reordering data; (q, d i , d j ) represents a triple randomly sampled from the reordering data set D; represents expectation of all triples sampled from the reordering data set D.

[0071] S3, obtaining a to-be-tested text query, preprocessing all API documents obtained in step S1, and obtaining an initial ordering API document recommendation sequence of the to-be-tested text query according to the to-be-tested text query and the preprocessed API documents.

[0072] S31, obtaining a to-be-tested text question and a corresponding code snippet, and splicing the to-be-tested text question and the corresponding code snippet to obtain a to-be-tested text query.

[0073] As shown in Figure 3 S32, mapping each API document obtained in step S1 into a document vector feature by using a vector retrieval model, and storing each API document and the corresponding document vector feature into a constructed vector retrieval library.

[0074] S33, mapping the to-be-tested text query into a to-be-tested text vector feature by using the vector retrieval model; calculating a cosine similarity value between the to-be-tested text vector feature and each document vector feature in the vector retrieval library, and selecting the first N API documents from the vector retrieval library in descending order of the cosine similarity values to obtain an initial ordering API document recommendation sequence of the to-be-tested text query.

[0075] The vector retrieval model uses a UnixCoder model.

[0076] In this embodiment, N is 50, that is, the obtained initial ordering API document recommendation sequence contains 50 API documents. As Figure 1As shown in the "API Recommendation Process," the top 50 API document sequences {torch.dot,bmm(),...,numpy.linalg.matrix_power} are retrieved through the text query q, corresponding to {d1,d2,...,d 50}

[0077] like Figure 1 As shown in step S4, the trained reordering model is used to process the initially ordered API document recommendation sequence and the text query to be tested to obtain the reordered API document recommendation sequence for the text query to be tested.

[0078] S41. Concatenate each API document in the initial sorted API document recommendation sequence to obtain N test text input data.

[0079] S42. Input the text input data to be tested into the trained reordering model for processing to obtain the relevance value of each text input data to be tested.

[0080] S43. Select the top M API documents from the initial sorted API document recommendation sequence based on relevance value from largest to smallest to obtain the reordered API document recommendation sequence for the text query to be tested.

[0081] In this embodiment, M is set to 10, meaning that the resulting reordered API document recommendation sequence contains 10 API documents.

[0082] Once the recommended sequence of reordered API documents is obtained, developers can perform searches based on each API document in the recommended sequence.

[0083] The API documentation at the top of the reordered recommendation sequence is the most recommended for developers, with the recommendation level decreasing for subsequent API documentation.

[0084] like Figure 1 As shown, an example developer's text query consists of the developer's text question and code snippets. A re-ranking model sequentially predicts the relevance of the top 50 API documents and re-ranks them to obtain the final top 10 API documents: {numpy.linalg.matrix_power,torch.dot,....,bmm()}. Here, "numpy.linalg.matrix_power" satisfies the user's textual intent and is consistent with the NumPy library used in the corresponding code snippets, effectively recommending relevant API documents and reducing the time spent by the user searching for API documents.

[0085] The results of the method of this invention compared with existing re-ranking models for API document recommendation on a re-ranking dataset are as follows:

[0086] Reordering model Recall@10 NDCG@10 MRR@10 Relevance generation 0.1998 0.1306 0.1030 UPR 0.2064 0.1200 0.0838 PRP-Sliding 0.1251 0.1152 0.1120 The method 0.3236 0.1839 0.1248

[0087] Note:

[0088] (1) Relevance Generation, UPR and PRP-Sliding are commonly used relevance generation algorithms.

[0089] (2)Recall@10: Recall@10∈[0,1], the larger the value, the better the API documentation recommendation effect.

[0090] (3) NDCG@10: NDCG@10∈[0,1], the larger the value, the better the API documentation recommendation effect.

[0091] (4) MRR@10: MRR@10∈[0,1], the larger the value, the better the API documentation recommendation effect.

[0092] (5) Compared with Relevance Generation, the present invention improves the metrics Recall@10, NDCG@10 and MRR@10 by 12.38%, 5.33% and 2.18% respectively on the same dataset.

[0093] (6) Compared with UPR, the present invention improves the metrics Recall@10, NDCG@10 and MRR@10 by 11.72%, 6.39% and 4.10% respectively on the same dataset.

[0094] (7) Compared with PRP-Sliding, the present invention improves the metrics Recall@10, NDCG@10 and MRR@10 by 19.85%, 6.87% and 1.28% respectively on the same dataset.

[0095] The method of this invention understands the query intent of developers and automatically recommends API documents related to development tasks, thereby effectively reducing the time developers spend looking up information and improving development efficiency.

[0096] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. An API document recommendation method based on a large language model, characterized in that, Includes the following steps: S1. Obtain several text queries and several API documents. Concatenate each text query and each API document to obtain text input data. Preset a relevance value for each text input data. Combine the text input data and the corresponding relevance value to form reordered data. Summarize all reordered data to obtain a reordered dataset. S2. Construct a re-ranking model. Use the text input data of the re-ranked data as input and the corresponding relevance value as label. Input the re-ranked dataset into the re-ranking model for training to obtain a trained re-ranking model. S3. Obtain the text query to be tested, preprocess all API documents obtained in step S1, and obtain the initial sorted API document recommendation sequence based on the text query to be tested and the preprocessed API documents. S4. The trained reordering model is used to process the initially ordered API document recommendation sequence and the text query to be tested to obtain the reordered API document recommendation sequence for the text query to be tested.

2. The API document recommendation method based on a large language model according to claim 1, characterized in that, Step S1 specifically involves: S11. Obtain several text questions from the developer and the code snippet corresponding to each text question. Concatenate each text question and the corresponding code snippet to obtain the developer's text query. S12. Collect several API documents, concatenate each text query with each API document to obtain text input data, preset the correlation value between the corresponding text query and API document for each text input data, combine the text input data and the corresponding correlation value as re-sorted data, and summarize all re-sorted data to obtain the re-sorted dataset.

3. The API document recommendation method based on a large language model according to claim 1, characterized in that, Its features are: The re-ranking model includes a natural language extractor, an inference state extractor, and a relevance detector, which are sequentially connected. The natural language extractor is a pre-trained natural language model and a fully connected layer. The inference state extractor consists of multiple sequentially connected state extraction layers. The re-ranked dataset is output to the input of the pre-trained natural language model for processing. The output of the natural language model is input to the fully connected layer for processing. The output of the fully connected layer is sequentially input to multiple sequentially connected state extraction layers for processing. The output of the last state extraction layer is input to the relevance detector for processing. The output of the relevance detector is used as the output of the re-ranking model.

4. The API document recommendation method based on a large language model according to claim 3, characterized in that, Its features are: Each of the state extraction layers is configured according to the following formula: s = LayerNorm(FFN(p″) + p″) p″=CrossAttention(p′,h′,h′) p′=SelfAttention(p,p,p) Where s represents the output of the state extraction layer; LayerNorm() represents the normalization layer; FFN() represents the feedforward network layer; CrossAttention() represents the cross-attention mechanism layer; SelfAttention() represents the self-attention mechanism layer; p represents the learning vector; h′ represents the output of the layer above the current state extraction layer; p′ represents the output of the self-attention mechanism; and p″ represents the output of the cross-attention mechanism layer.

5. The API document recommendation method based on a large language model according to claim 3, characterized in that, Its features are: The correlation detector is set according to the following formula: r = W * g″ + b g″=LayerNorm(g′+FFN(g′)) g′=LayerNorm(MHA(g,s,s)) Where r represents the output of the correlation detector, which is also the correlation value; W represents the learning weight; b represents the bias; LayerNorm() represents the normalization layer; FFN() represents the feedforward network layer; MHA() represents the multi-head attention mechanism layer; g represents the learning vector; s represents the output of the last state extraction layer in the inference state extractor; g′ represents the output of the first normalization layer in the correlation detector; and g″ represents the output of the second normalization layer in the correlation detector.

6. The API document recommendation method based on a large language model according to claim 3, characterized in that, Its features are: The loss function of the reordering model is set according to the following formula: Where Loss represents the loss function of the reordering model; K represents the number of reordered data in the training batch; q represents the expectation; q represents the text problem in reordering data; i and j both represent indices; d i and d j Let d represent the API documents in the i-th reordered dataset and the j-th reordered dataset, respectively; D represents the reordered dataset; σ represents the logistic function; r represents the relevance value; and d represents the API documents in the i-th reordered dataset. i The corresponding relevance value is greater than that of the API document d for the j-th reordered data. j The corresponding correlation values; (q,d) i ,d j ) represents a triple randomly sampled from the reordered dataset D; This represents the expectation of all triples sampled from the reordered dataset D.

7. The API document recommendation method based on a large language model according to claim 1, characterized in that, Step S3 specifically involves: S31. Obtain the text question to be tested and the corresponding code snippet, and concatenate the text question to be tested and the corresponding code snippet to obtain the text query to be tested; S32. Using a vector retrieval model, each API document obtained in step S1 is mapped to a document vector feature, and each API document and its corresponding document vector feature are stored in the constructed vector retrieval library. S33. Use a vector retrieval model to map the text query to be tested into the vector features of the text to be tested; calculate the cosine similarity value between the vector features of the text to be tested and the vector features of each document in the vector retrieval library; select the top N API documents from the vector retrieval library in descending order of cosine similarity value to obtain the initial sorted API document recommendation sequence for the text query to be tested.

8. The API document recommendation method based on a large language model according to claim 7, characterized in that: The vector retrieval model adopts the UnixCoder model.

9. The API document recommendation method based on a large language model according to claim 1, characterized in that, Step S4 specifically involves: S41. Concatenate each API document in the API document recommendation sequence of the text query to be tested and the initial sorted API document to obtain N text input data to be tested. S42. Input the text input data to be tested into the trained reordering model for processing, and obtain the relevance value of each text input data to be tested. S43. Select the top M API documents from the initial sorted API document recommendation sequence based on relevance value from largest to smallest to obtain the reordered API document recommendation sequence for the text query to be tested.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images