A research and development system based on an auxiliary large model and a research and development auxiliary framework

Abstract

A research and development system based on an auxiliary large model and a research and development auxiliary framework, the system comprises: a code auxiliary large model, receiving a code auxiliary request, providing code generation, completion, and explanation basic capabilities; the system further comprises: a research and development auxiliary framework system, which is used for fusing the code auxiliary large model and the CREATE research and development auxiliary framework, and realizes intelligent support for the whole process from requirement understanding to knowledge sedimentation; wherein each decoding block calculation sublayer comprises four submodel layers of a multi-head self-attention mechanism layer, a residual connection, an intermediate process layer normalization, and a feedforward neural network module. The present application has the advantages of improving the research and development efficiency of the system, improving the code research and development quality, and reducing the software maintenance cost.

Description

Technical Field

[0001] This invention relates to the field of R&D system technology, and in particular to an R&D system based on an auxiliary large model and an R&D auxiliary framework. Background Technology

[0002] This invention relates to intelligent support for traditional software development tools and models in commercial banks, aiming to improve problems such as long development cycles, inconsistent code quality, and difficulties in knowledge transfer during the development process. This will help developers improve system development efficiency, code quality, and reduce software maintenance costs, and facilitate the standardization and intelligent transformation of commercial banks' financial technology development process.

[0003] Current problems:

[0004] First, with the development, innovation, and digital transformation of commercial banking operations, the application of financial technology in commercial banking systems is becoming increasingly widespread. This places higher demands on R&D efficiency and quality in areas such as product development, system maintenance, and business process optimization. Traditional commercial banking software development tools and models, when faced with challenges such as complex business requirements, rapid iteration cycles, and the stability of high-concurrency systems, are gradually revealing problems such as long development cycles, inconsistent code quality, and difficulties in knowledge transfer.

[0005] Second, currently, commercial bank R&D personnel lack the support of intelligent tools in the development of financial-related business products and systems.

[0006] Third, current mainstream technical solutions and commercial banking practices typically lack an integrated, intelligent R&D support framework that covers the entire lifecycle of code development (the entire chain of requirements understanding, development, maintenance, testing, and knowledge accumulation); and there is a general lack of an effective knowledge management and accumulation mechanism for reusable financial business code assets (i.e., the "RAG of Code" code knowledge base module in the CREATE intelligent R&D support framework), which makes it difficult to systematically accumulate and reuse valuable financial business code knowledge.

[0007] Furthermore, against the backdrop of commercial banks accelerating the development of digital finance and focusing on empowering the construction of the digital economy, commercial bank R&D personnel relatively lack the support of intelligent tools in the process of developing fintech-related business products and systems. Summary of the Invention

[0008] The purpose of this invention is to provide a research and development system based on an auxiliary large model and a research and development support framework, which has the advantage of intelligence.

[0009] To achieve the above objectives, this invention provides a research and development system based on a large-scale auxiliary model and a research and development support framework. The system includes: a large-scale code-assisted model that receives code-assisted requests and provides basic capabilities for code generation, completion, and interpretation; the large-scale code-assisted model includes: a token encoding and position encoding module, which learns the relative positional information between characters or words; an initial layer normalization module that normalizes the input vector, making the intermediate results of training and inference more stable and robust; and a decoding block computation layer, which is the specific computation layer of the large-scale code-assisted model, formed by stacking multiple decoding block computation sub-layers; the token encoding and position encoding module, the initial layer normalization module, and the decoding block computation layer communicate with each other; the system also includes: a research and development support framework system, which integrates the large-scale code-assisted model and the CREATE research and development support framework, realizing intelligent support throughout the entire process from requirement understanding to knowledge accumulation; wherein each decoding block computation sub-layer contains four sub-model layers: a multi-head self-attention mechanism layer, residual connections, intermediate process layer normalization, and a feedforward neural network module.

[0010] Preferably, the Token encoding and position encoding module is used to learn the relative position information between characters or words; the Token encoding and position encoding module receives the "Token encoding" vector and the "position encoding" to obtain the initial input vector H(0):

[0011] ;

[0012] Among them, TokenEmbedding is the vectorized encoding representation of "characters" and "words" formed after word segmentation of natural language description text or code such as requirements, problems or instructions related to programming development by R&D personnel, after being encoded by a word embedding model; its essence is to convert natural language text or code into vector encoding form, so as to facilitate input code to assist large models in calculation; PositionEmbedding is the vectorized encoding representation of the relative or absolute position order of "characters" and "words" formed after word segmentation of natural language description text or code. RoPE is a relative position encoding algorithm that enables code to assist large models in learning the relative position information between characters or words.

[0013] Preferably, the initial layer normalization module is used to stabilize the code-assisted training process of large models. The computational layer of the initial layer normalization module consists of multiple decoding block computational layers, and its calculation formula is as follows:

[0014] ;

[0015] Among them, H （l） normH is the normalized hidden state vector output after layer normalization computation of the input hidden state vectors of the Token encoding and position encoding modules; g is a learnable scaling parameter, ε is a minimal constant to prevent division by zero; mean represents the average value of all input vectors, and l represents the total number of computation layers of the initial layer normalization module, l=1, 2...80; when l is 1 to 80 layers, H （1） That is, the input hidden state vector H of the (l-1)th computational layer. （l-2） norm After layer normalization calculation, the output is the normalized hidden state vector H of the (l-1)th layer. （l-1） norm That is, the upper-level decoding block computation layer is the computation input for the lower-level decoding block computation layer; RMSNorm represents normalized computation.

[0016] Preferably, the calculation formula for the decoding block computation layer is as follows:

[0017] ; ;

[0018] Where Attn is the multi-head self-attention mechanism computation process; FFN is the feed-through neural network computation process; W vocab To compute the output hidden state vector H of the 80th layer decode block. （80） The "trainable weight parameter matrix" is required when performing the nonlinear projection transformation calculation of the "candidate output word space"; Output is the "candidate output word space" vector calculated by the code-assisted large model. By using different sampling strategies for the "candidate output word space" vector, the final character or word generated by the code-assisted large model at each step can be determined.

[0019] Preferably, in the multi-head self-attention mechanism layer, multi-head self-attention mechanism calculation is performed, and the calculation includes:

[0020] Vector calculation of intermediate processes for Query, Key, and Value:

[0021] ;

[0022] Among them, W q （l） W k （l） With W v （l） These are the "trainable weight parameter matrices" used in the vector calculation of the intermediate processes of Query, Key, and Value in layer l, respectively; d l d is the number of feature dimensions of the hidden state vector; head The number of feature dimensions for each self-attention head vector; the normalized hidden state vector H for each layer. （l）norm In the multi-head self-attention mechanism, the nonlinear projection transformation computation is split into h self-attention head vectors; h represents the number of self-attention heads, that is, the number of attention heads computed in parallel in multi-head self-attention, which is used to split the normalized hidden state vector into multiple subspaces for independent computation; R is the symbol of the set of real numbers, which in this case is the symbolic representation of the relevant parameter matrix and its shape.

[0023] In the multi-head self-attention mechanism layer, the positional encoding based on RoPE is used to calculate the relative positional information during the multi-head self-attention mechanism calculation process, as follows:

[0024] ;

[0025] Where Q represents the query vector, and in the self-attention mechanism, Q represents the vector of other words currently being "attended"; it is usually obtained by multiplying the input word vector by a learnable weight matrix Wq; K represents the key vector; K represents the query word vector, which is obtained by multiplying the input word vector by another learnable weight matrix Wk; Q （1） This represents the query vector after applying rotational position encoding, which is the core operation object of RoPE; it does not simply "add" the position vector to Q, but rather performs a geometric rotation on Q using a rotation matrix to include absolute position information; K （1） This represents the key vector after applying rotational position encoding; similarly, K also undergoes the same rotation operation to incorporate the corresponding position information.

[0026] Preferably, the multi-head self-attention mechanism layer further includes: attention score calculation:

[0027] ;

[0028] The numerator is the scaling dot product calculation, which is used to stabilize gradient values ​​during the training of large models, thereby improving the training effect. Mask is an upper triangular attention mask, used to ensure that the current position only focuses on itself and previous historical tokens. Softmax is a mathematical function that maps a vector to a probability distribution, ensuring that all probabilities are greater than 0 and the sum of probabilities equals 1. Attention represents the attention mechanism, which is used to calculate the correlation between each token and other tokens. Mask is the calculation process to ensure that the current position only focuses on itself and previous historical tokens. T represents the vector transpose operation.

[0029] Preferably, the self-attention head calculation is as follows:

[0030] ;

[0031] Based on the above calculation formula, a total of h self-attention head vectors are calculated; V represents the value matrix, which is the "attention score matrix Attention(l)" used for the "dot product" calculation when calculating the "self-attention head vector head"; l represents the number of the "decoding block computation layer" from layer 1 to layer 80. Since the "multi-head self-attention mechanism" is a sub-computation process in the "decoding block computation layer", each l-layer "decoding block computation layer" will also contain a "multi-head self-attention mechanism" calculation process; in the "self-attention head calculation" process, the input sequence is multiplied by the query matrix Q, the key matrix K, and the value matrix V respectively, and then they are concatenated according to certain rules. Finally, a linear transformation is performed to obtain the vector result value of the output sequence.

[0032] Output results calculated by the multi-head self-attention mechanism

[0033] ;

[0034] Where Concat is the horizontal concatenation calculation of h self-attention head vectors; W o （l） This is the trainable weight parameter matrix for multi-head self-attention vectors during nonlinear projection transformation output computation. In the "multi-head self-attention mechanism", h self-attention head vectors can be computed in parallel, and this parallel computation process can significantly enhance the efficiency of code-assisted large model training and inference.

[0035] Preferably, the calculation is performed twice in each "decoding block calculation layer", and the calculation includes:

[0036] (1) Residual connections after multi-head self-attention mechanism computation;

[0037] ;

[0038] Among them, H （l） The input hidden state vector of the layer is calculated for the l-th decoder block; AttOut （l） This is the output result calculated by the multi-head self-attention mechanism;

[0039] (2) Residual connections after feedforward neural network computation;

[0040] ;

[0041] Among them, FFNOut （l） The output result calculated by the feedforward neural network; H （l+1） Calculate the output hidden state vector of the layer for the l-th decoding block;

[0042] (3) Normalized calculation:

[0043] The intermediate process of "layer normalization" targets the calculation result Z of the "residual connectivity after multi-head self-attention mechanism". （l） norm The normalization calculation is performed using the following formula:

[0044] ;

[0045] When l is 1 to 80 layers, Z （1） The result vector calculated by the "residual connection" model layer in the computation layer of the l-th decoding block; H （l） norm When l is from 1 to 80, Z(l) is the normalized result vector output by the "residual connection" model layer in the l-th decoding block computation layer after "layer normalization" calculation; Mean is the average value of all input vectors in the layer normalization calculation formula, and "l" represents the l-th decoding block computation layer.

[0046] Preferably, the feedforward neural network module is the core nonlinear projection transformation calculation process of each "decoding block calculation layer", and its input is the normalized calculation result of the intermediate process layer. The calculation formula is as follows:

[0047] ;

[0048] SwiGLU is a non-linear activation function used to help the code assist large models in performing non-linear fitting and learning of training data, which can significantly enhance the expressive power of the model; W(l)down, W(l)up, and W(l)gate are weight vector matrices required for performing "vector matrix multiplication" in the "feedforward neural network" model layer in the l-th decoding block computation layer. This represents "dot multiplication"; Swiglu is a non-linear activation function that combines the ideas of the Swish function and gated linear units.

[0049] Preferably, the R&D support framework system includes: a code completion and generation module for completing and generating R&D code that meets requirements; a code knowledge base module for storing, retrieving, and accumulating R&D knowledge assets; a code explanation module for generating code explanation, review, and optimization solutions; a code commenting module for generating compliant and maintainable code comments; a unit test generation module for generating high-coverage compliant test cases; and a code Q&A module for answering technical and business logic questions. The code completion and generation module, code knowledge base module, code explanation module, code commenting module, unit test generation module, and code Q&A module are directly interconnected in both directions.

[0050] In summary, compared with the prior art, the R&D system based on an auxiliary large model and an R&D auxiliary framework provided by the present invention has the following beneficial effects:

[0051] This system not only helps commercial banks' fintech R&D personnel improve system development efficiency, code development quality, and reduce software maintenance costs, but also assists in the standardization and intelligent transformation of commercial banks' fintech R&D process, providing an explainable, sustainable, and evolving solution for fintech R&D. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of a research and development system based on an auxiliary large model and a research and development auxiliary framework proposed in this invention. Detailed Implementation

[0053] The following will be combined with the appendix in the embodiments of the present invention. Figure 1 The technical solutions, structural features, objectives and effects achieved in the embodiments of the present invention will be described in detail.

[0054] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions. They are only used to facilitate and clarify the purpose of illustrating the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationship, or adjustments to the size should still fall within the scope of the technical content disclosed in the present invention, provided that they do not affect the effects and objectives that the present invention can produce.

[0055] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0056] This invention proposes a research and development system based on a large-scale auxiliary model and a research and development support framework. Specifically, this invention innovatively proposes an "intelligent research and development method and system for commercial banks based on a large-scale code-assisted model and a CREATE research and development support framework." This intelligent research and development method integrates the key technologies of the "CREATE research and development support framework," such as the "large-scale code-assisted model," code completion and generation, code knowledge base (RAG), code explanation, code annotation, generating unit tests (Testing Automation of Code), and code question answering (Enquiry of Code), to construct an intelligent research and development support system for commercial banks.

[0057] Code-assisted large-scale models are massive language models based on ultra-large-scale Transformer-Decoder architectures, typically possessing billions to hundreds of billions of parameters. The core capabilities of these models, such as code assistance and generation, primarily stem from single-stage or multi-stage autoregressive pre-training processes performed on large-scale, multilingual, high-quality code corpora using distributed training frameworks. This pre-training process often relies on computing clusters composed of hundreds to thousands of GPUs, leveraging key technologies such as parallel computing, high-speed interconnects, mixed precision, and ZeRO to overcome computational and communication bottlenecks. Furthermore, after pre-training, the code-assisted large-scale model undergoes fine-tuning training through processes such as instruction fine-tuning (SFT) and human preference alignment (RLHF / DPO / DAPO) to better understand and follow the needs and instructions of developers. Simultaneously, during inference, the code-assisted large-scale model employs an autoregressive generation mechanism, combined with sampling strategies, temperature control, and acceleration techniques such as key-value caching and quantization, achieving efficient and controllable code assistance and generation. The massive parameters of code-assisted large-scale models endow them with powerful pattern recognition, logical reasoning, and long-context understanding capabilities, enabling them to learn various complex programming syntaxes, semantics, library / API usage patterns, and the mapping relationships between code and natural language descriptions. However, the larger the number of parameters in a code-assisted large-scale model, the more training code corpus data is required. Therefore, when constructing the code corpus dataset, it is necessary to collect large-scale, high-quality, multi-language (such as Python, Java, Vue, JavaScript, SQL, etc.) "public source code" and "corresponding natural language text" (such as code comments, code problem descriptions, official technical documentation, R&D requirement documents, etc.) from multiple channels such as "GitHub," "StackOverflow," "official technical documentation," and "programming domain benchmark datasets." The collected corpus data must undergo rigorous cleaning to remove low-quality, insecure, and copyright-infringing content, and deduplication operations are performed. At the same time, the code language and code task type are balanced to ultimately construct a dedicated code corpus dataset focused on training code-assisted large-scale models in the programming domain.

[0058] Based on the foregoing, such as Figure 1 As shown, the system includes:

[0059] The code assistance model receives code assistance requests and provides basic capabilities such as code generation, completion, and explanation.

[0060] The code-assisted large model includes:

[0061] The Token encoding and position encoding module is used to learn the relative position information between characters or words;

[0062] The initial layer normalization module normalizes the input vector, making the intermediate results of training and inference more stable and robust.

[0063] The decoding block computation layer is the specific computation layer of the code-assisted large model, which is formed by stacking multiple decoding block computation sub-layers.

[0064] Data communication is established between the Token encoding and position encoding module, the initial layer normalization module, and the decoding block calculation layer;

[0065] The system also includes: a research and development support framework system, which integrates the code support model and the CREATE research and development support framework to achieve intelligent support for the entire process from requirement understanding to knowledge accumulation;

[0066] Each decoding block computational sublayer contains four sub-model layers: a multi-head self-attention mechanism layer, a residual connection, an intermediate process layer normalization, and a feedforward neural network module (equivalent to each "decoding block computational sublayer being constructed by stacking these four sub-model layers one after another. This will be explained in detail in the subsequent instructions).

[0067] Specifically, the Token encoding and position encoding module is used to learn the relative position information between characters or words; the Token encoding and position encoding module receives the "Token encoding" vector and the "position encoding" to obtain the initial input vector H(0):

[0068] ;

[0069] Among them, TokenEmbedding is a vectorized encoding representation of the "characters" (code) and "words" (text) formed after word segmentation of natural language description text or code related to R&D needs, problems or instructions, and then encoded by the word embedding model. Its essence is to convert natural language text or code into vector encoding form, so as to facilitate the input code to assist large models in calculation. PositionEmbedding is a vectorized encoding representation of the relative or absolute position order of "characters" and "words" formed after word segmentation of natural language description text or code. RoPE (Rotary Position Embedding) is a relative position encoding algorithm that enables code to assist large models in learning the relative position information between characters or words.

[0070] The initial layer normalization module is used to stabilize the code and assist in the training process of large models. As mentioned earlier, the computation layer of the initial layer normalization module consists of multiple decoding block computation layers, and its calculation formula is as follows:

[0071] ;

[0072] Among them, H （l） norm H is the normalized hidden state vector output after layer normalization computation of the input hidden state vectors of the Token encoding and position encoding modules; g is a learnable scaling parameter, ε is a minimal constant to prevent division by zero; mean represents the average value of all input vectors, and l represents the total number of computation layers of the initial layer normalization module, l=1, 2...80; when l is 1 to 80 layers, H （1） That is, the input hidden state vector H of the (l-1)th computational layer. （l-2） norm After layer normalization calculation, the output is the normalized hidden state vector H of the (l-1)th layer. （l-1） norm That is, the computation output of the previous decoder block (Decoder Block) serves as the computation input for the next decoder block (Decoder Block); RMSNorm represents normalized computation. The aforementioned formula normalizes the input vector, ensuring that the vector values ​​in each dimension are on the same order of magnitude, which helps improve the training performance of large models assisted by code.

[0073] The following will describe the specific process of the decoding block computation layer.

[0074] The calculation process of the decode block computation layer can be represented by the following formula:

[0075] ; ;

[0076] Where Attn is the multi-head self-attention mechanism computation process; FFN is the feed-through neural network computation process; W vocab The output hidden state vector H for the 80th layer decoder block computation layer. （80） The "trainable weight parameter matrix" is required when performing the nonlinear projection transformation calculation of the "candidate output word space"; Output is the "candidate output word space" vector calculated by the code-assisted large model. By using different sampling strategies for the "candidate output word space" vector, the final character or word generated by the code-assisted large model at each step can be determined.

[0077] Specifically, in the multi-head self-attention mechanism layer, a multi-head self-attention mechanism calculation is performed, the calculation including:

[0078] Vector calculation of intermediate processes for Query, Key, and Value:

[0079]

[0080] Among them, W q （l） W k （l） With W v （l） These are the "trainable weight parameter matrices" used in the vector calculation of the intermediate processes of Query, Key, and Value in layer l, respectively; d l d is the number of feature dimensions of the hidden state vector; head The number of feature dimensions for each self-attention head vector; the normalized hidden state vector H for each layer. （l） norm In the multi-head self-attention mechanism, the nonlinear projection transformation computation is split into h self-attention head vectors; h represents the number of self-attention heads, that is, the number of attention heads computed in parallel in multi-head self-attention, which is used to split the normalized hidden state vector into multiple subspaces for independent computation; R is the symbol of the set of real numbers, which in this case is the symbolic representation of the relevant parameter matrix and its shape.

[0081] In the multi-head self-attention mechanism layer, the positional encoding calculation based on RoPE can maintain the relative positional information during the multi-head self-attention mechanism calculation process.

[0082]

[0083] Here, Q represents the query vector. In the self-attention mechanism, Q represents the vector of other tokens currently being "attended"; it is typically obtained by multiplying the input word vectors (embeddings) by a learnable weight matrix Wq. K represents the key vector; K represents the query token vector, which is obtained by multiplying the input word vectors by another learnable weight matrix Wk. （1） This represents the query vector after applying rotational position encoding, which is the core operation object of RoPE. It doesn't simply "add" the position vector to Q; instead, it geometrically rotates Q using a rotation matrix to include absolute position information. K （1） This represents the key vector after applying rotational position encoding. Similarly, K also undergoes the same rotation operation to incorporate the corresponding position information.

[0084] The multi-head self-attention mechanism layer also includes: Scaled Dot-ProductAttention Score calculation:

[0085]

[0086] The numerator is a scaled dot product calculation, used to stabilize gradient values ​​during large model training, improving training performance. The Mask is an upper triangular attention mask used to ensure that the current position only focuses on itself and previous historical tokens. Softmax is a mathematical function that maps a vector to a probability distribution, ensuring all probabilities are greater than 0 and the sum of probabilities equals 1. Attention represents the attention mechanism, used to calculate the correlation between each token and other tokens. The Mask is the calculation process to ensure that the current position only focuses on itself and previous historical tokens. T represents vector transpose. The Scaled Dot-Product Attention Score is calculated to determine which tokens should be assigned higher weights to better capture important information in the input sequence. In other words, it finds important tokens in the input sequence and weights them, thereby improving the performance of code-assisted large models.

[0087] In a specific embodiment, the self-attention head is calculated as follows:

[0088]

[0089] Based on the above calculation formula, a total of h self-attention head vectors are calculated; V represents the value matrix, which is the "attention score matrix Attention(l)" used for the "dot product" calculation when calculating the "self-attention head vector head". l represents the number of layers of the "Decoder Block" from layer 1 to layer 80. Because the "multi-head self-attention mechanism" is a sub-computation process in the "Decoder Block", each l-layer "Decoder Block" will also contain a "multi-head self-attention mechanism" calculation process. The role of the "self-attention head" calculation process is to calculate and compare the attention scores of each vector element in the input sequence with other vector elements to obtain the relationship between them, and integrate these relationships to obtain the representation of each vector element. This allows the code to help the large model better capture the semantic information in the input sequence and improve the model's performance. Specifically, it multiplies the input sequence with the query matrix Q, the key matrix K, and the value matrix V respectively (that is, it needs to calculate the similarity of each vector element with other vector elements, and then sum them in weights based on these similarities), then concatenates them according to certain rules, and finally performs a linear transformation to obtain the vector result value of the output sequence.

[0090] Output results calculated by the multi-head self-attention mechanism

[0091]

[0092] Where Concat is the horizontal concatenation calculation of h self-attention head vectors; W o （l） This is the trainable weight parameter matrix for multi-head self-attention vectors during nonlinear projection transformation output computation. In the "multi-head self-attention mechanism", h self-attention head vectors can be computed in parallel, and this parallel computation process can significantly enhance the efficiency of code-assisted large model training and inference.

[0093] Specifically, the residual connection serves to alleviate overfitting of the training data during fine-tuning training of large models, thereby improving the generalization performance of large models and their question-answering effectiveness.

[0094] Each "Decoder Block" layer is calculated twice, and the calculation includes:

[0095] (1) Residual connections after multi-head self-attention mechanism computation;

[0096]

[0097] Among them, H （l） AttOut is the input hidden state vector of the l-th decoder block computation layer; （l） This is the output result calculated by the multi-head self-attention mechanism;

[0098] (2) Residual connections after feedforward neural network computation;

[0099]

[0100] Among them, FFNOut （l） The output result calculated by the feedforward neural network; H （l+1） This is the output hidden state vector of the l-th decoder block computation layer (also the input hidden state vector of the (l+1)-th decoder block computation layer (Decoder Block)).

[0101] (3) Normalized calculation:

[0102] The intermediate process "LayerNorm" calculates the result Z of the residual connections after multi-head self-attention mechanism computation. （l） norm The normalization calculation is performed using the following formula:

[0103]

[0104] When l is 1 to 80 layers, Z （1） The result vector is calculated for the "Residual Connection" model layer in the L-th decoder block computation layer. H （l） norm When l ranges from 1 to 80, Z(l) is the output vector of the residual connection model layer in the l-th decoder block computation layer. After layer normalization, it becomes the normalized output vector. "Mean" in the layer normalization formula represents the average value of all input vectors, and "l" represents the l-th decoder block computation layer. This formula normalizes the input vectors, ensuring that the vector values ​​in each dimension are on the same order of magnitude, which improves the training performance of large code-assisted models.

[0105] Specifically, the feedforward neural network module is the core nonlinear projection transformation calculation process of each "Decoder Block" layer, and its input is the normalized calculation result of the intermediate process layer. The calculation formula is as follows:

[0106]

[0107] SwiGLU is a non-linear activation function used to help the code assist large models in performing non-linear fitting and learning of training data, which can significantly enhance the expressive power of the model; W(l)down, W(l)up, and W(l)gate are weight vector matrices used to perform vector matrix multiplication operations in the "feedforward neural network" model layer in the l-th layer of the decoder block computation layer. This indicates "dot multiplication"; SwiGLU is a non-linear activation function, short for "Swish-Gated Linear Unit," which combines the ideas of the Swish function and the Gated Linear Unit (GLU). It is used to help code assist large models in performing non-linear fitting and learning on training data, significantly enhancing the model's expressive power.

[0108] The R&D support framework system includes:

[0109] The code completion and generation module completes and generates development code that meets specific requirements.

[0110] The code knowledge base module stores, retrieves, and consolidates R&D knowledge assets.

[0111] The code explanation module generates code explanations, reviews, and optimization plans.

[0112] The code comment module generates compliant and maintainable code comments;

[0113] Generate unit test modules to generate high-coverage compliant test cases;

[0114] The code Q&A module answers technical and business logic questions;

[0115] The code completion and generation module, code knowledge base module, code explanation module, code comment module, unit test generation module, and code Q&A module are directly interconnected in both directions.

[0116] (1) Data flow in code completion scenarios:

[0117] When developers input text data such as "requires complete development code" or "development code with missing intermediate content," the system will simultaneously input text data such as "code continuation instruction" or "code completion instruction." At this time, the "code completion and generation" module will integrate the "development code context content" input by the developer and the relevant "code continuation instruction" or "code completion instruction" into a complete text data as input to the "code assistance model." After receiving the complete text data, the "code assistance model" will perform the code completion operation and output the completed development code.

[0118] (2) Data flow in code generation scenarios:

[0119] When R&D personnel input natural language descriptive text data related to "R&D requirements," the system will simultaneously input text data for "code generation instructions." The "Code Completion and Generation" module will integrate the R&D personnel's "R&D requirements" and "code generation instructions" into a complete text data set as input to the "Code Assistance Model." After receiving the complete text data, the "Code Assistance Model" will execute the code generation operation and output the generated R&D code.

[0120] In a specific embodiment, the code completion and generation module includes a financial business code knowledge embedding preprocessing module. In the "financial business code knowledge embedding preprocessing module", the original files related to the code assets and knowledge in the commercial bank's financial business domain (such as financial business R&D requirement documents, R&D specification documents, R&D process documents, R&D code engineering files, etc.) are segmented into different "R&D knowledge fragments". After the segmentation of the "R&D knowledge fragments" is completed, the "EmbeddingModel" encodes the different "R&D knowledge fragments" into searchable structured vectors. Finally, the "structured vectors" and "metadata information" of all "R&D knowledge fragments" are stored in a vector database (such as Milvus).

[0121] Code explanation module,

[0122] The intelligent R&D assistance capability of code explanation is built upon the code parsing, code review and optimization capabilities of a large-scale code assistance model, as well as a code knowledge base.

[0123] During the intelligent code parsing process, when developers input the "development code to be parsed (such as core transaction logic code, risk control algorithm code, etc.)" and the "code parsing requirements," the system will input "code parsing instructions." Simultaneously, the system automatically connects to the "code knowledge base," retrieving code examples from similar financial business scenarios, internal compliance documents, internal technical specifications, historical parsing records, and other development knowledge fragments based on the developers' input of the "development code to be parsed" and "code parsing requirements." This allows the system to construct enhanced contextual code parsing instructions for the financial business domain. Finally, the system integrates the "development code to be parsed," the "code parsing requirements," and the "enhanced contextual code parsing instructions for the financial business domain" and inputs them into the code assistance model. The code assistance model then provides a more detailed, accurate, and precisely tailored code parsing response that aligns with the logic of commercial bank financial business.

[0124] Generate unit test module

[0125] It provides intelligent R&D support capabilities for generating unit tests (Testing Automation of Code), and builds unit test schemes and test cases based on a large code-assisted model, as well as a code knowledge base.

[0126] In the intelligent assistance process for generating unit tests, after developers determine the development plan and submit the development code, and testers input the "test requirements and plan," the system automatically connects to the "code knowledge base" to retrieve historical test cases, internal testing specification documents, and financial business logic rule description documents in similar financial business development scenarios. This allows the system to construct Financial Business Enhancement Contextual Testing Instructions. Finally, the system integrates the "development plan," "test requirements and plan," and "financial business enhancement test contextual instructions" and inputs them into the code assistance model. Based on the development and testing context, the code assistance model generates unit test cases with high coverage and strong compliance, including financial business logic tests, boundary value tests, exception tests, and performance tests.

[0127] Code Q&A Module

[0128] Enquiry of Code, a key closed-loop capability of the CREATE R&D support framework, leverages multi-round semantic parsing, contextual reasoning, question answering, and knowledge generalization capabilities based on a large code-assisted model. Combined with the enhanced knowledge retrieval mechanism of the RAGof Code knowledge base within the financial business domain, it provides commercial bank R&D personnel with precise and intelligent question answering support covering financial business logic, technical specifications, and historical cases.

[0129] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A research and development system based on an auxiliary large model and a research and development auxiliary framework, characterized in that, The system includes: a large code assistance model that receives code assistance requests and provides basic capabilities for code generation, completion, and explanation; The code-assisted large model includes: The Token encoding and position encoding module is used to learn the relative position information between characters or words; The initial layer normalization module normalizes the input vector, making the intermediate results of training and inference more stable and robust. The decoding block computation layer is the specific computation layer of the code-assisted large model, which is formed by stacking multiple decoding block computation sub-layers. Data communication is established between the Token encoding and position encoding module, the initial layer normalization module, and the decoding block calculation layer; The system also includes: The development support framework system integrates the code support model and the CREATE development support framework, enabling intelligent support for the entire process from requirement understanding to knowledge accumulation. Each decoding block computational sub-layer contains four sub-model layers: a multi-head self-attention mechanism layer, a residual connection layer, an intermediate process layer normalization layer, and a feedforward neural network module.

2. The R&D system based on an auxiliary large model and an R&D auxiliary framework according to claim 1, characterized in that, The Token encoding and position encoding module is used to learn the relative position information between characters or words; the Token encoding and position encoding module receives the "Token encoding" vector and the "position encoding" to obtain the initial input vector H(0): ； Among them, TokenEmbedding is the vectorized encoding representation of "characters" and "words" formed after word segmentation of natural language description text or code such as requirements, problems or instructions related to programming development by R&D personnel, after being encoded by a word embedding model; its essence is to convert natural language text or code into vector encoding form, so as to facilitate input code to assist large models in calculation; PositionEmbedding is the vectorized encoding representation of the relative or absolute position order of "characters" and "words" formed after word segmentation of natural language description text or code. RoPE is a relative position encoding algorithm that enables code to assist large models in learning the relative position information between characters or words.

3. The R&D system based on an auxiliary large model and an R&D auxiliary framework according to claim 2, characterized in that, The initial layer normalization module is used to stabilize the code and assist in the training process of large models. The computation layer of the initial layer normalization module consists of multiple decoding block computation layers, and its calculation formula is as follows: ； Among them, H （l） norm H is the normalized hidden state vector output after layer normalization computation of the input hidden state vectors of the Token encoding and position encoding modules; g is a learnable scaling parameter, ε is a minimal constant to prevent division by zero; mean represents the average value of all input vectors, and l represents the total number of computation layers of the initial layer normalization module, l=1, 2...80; when l is 1 to 80 layers, H （1） That is, the input hidden state vector H of the (l-1)th computational layer. （l-2） norm After layer normalization calculation, the output is the normalized hidden state vector H of the (l-1)th layer. （l-1） norm That is, the upper-level decoding block computation layer is the computation input for the lower-level decoding block computation layer; RMSNorm represents normalized computation.

4. The R&D system based on an auxiliary large model and an R&D auxiliary framework according to claim 3, characterized in that, The calculation formula for the decode block computation layer is as follows: ； ； Where Attn is the multi-head self-attention mechanism computation process; FFN is the feed-through neural network computation process; W vocab To compute the output hidden state vector H of the 80th layer decode block. （80） The "trainable weight parameter matrix" is required when performing nonlinear projection transformation calculation of the "candidate output word space"; Output is the "candidate output word space" vector calculated by the code-assisted large model. By using different sampling strategies for the "candidate output word space" vector, the final character or word generated by the code-assisted large model at each step can be determined.

5. The R&D system based on an auxiliary large model and an R&D auxiliary framework according to claim 4, characterized in that, In the multi-head self-attention mechanism layer, multi-head self-attention mechanism calculations are performed, including: Vector calculation of intermediate processes for Query, Key, and Value: ； Among them, W q （l） W k （l） With W v （l） These are the "trainable weight parameter matrices" used in the vector calculation of the intermediate processes of Query, Key, and Value in layer l, respectively; d l d is the number of feature dimensions of the hidden state vector; head The number of feature dimensions for each self-attention head vector; the normalized hidden state vector H for each layer. （l） norm In the multi-head self-attention mechanism, the nonlinear projection transformation computation is split into h self-attention head vectors; h represents the number of self-attention heads, that is, the number of attention heads computed in parallel in multi-head self-attention, which is used to split the normalized hidden state vector into multiple subspaces for independent computation; R is the symbol of the set of real numbers, which in this case is the symbolic representation of the relevant parameter matrix and its shape. In the multi-head self-attention mechanism layer, the positional encoding based on RoPE is used to calculate the relative positional information during the multi-head self-attention mechanism calculation process, as follows: ； Where Q represents the query vector, and in the self-attention mechanism, Q represents the vector of other words currently being "attended"; it is usually obtained by multiplying the input word vector by a learnable weight matrix Wq; K represents the key vector; K represents the query word vector, which is obtained by multiplying the input word vector by another learnable weight matrix Wk; Q （1） This represents the query vector after applying rotational position encoding, which is the core operation object of RoPE; it does not simply "add" the position vector to Q, but rather performs a geometric rotation on Q using a rotation matrix to include absolute position information; K （1） This represents the key vector after applying rotational position encoding; similarly, K also undergoes the same rotation operation to incorporate the corresponding position information.

6. The R&D system based on an auxiliary large model and an R&D auxiliary framework according to claim 5, characterized in that, The multi-head self-attention mechanism layer also includes: attention score calculation: ； The numerator is the scaling dot product calculation, which is used to stabilize gradient values ​​during the training of large models, thereby improving the training effect. Mask is an upper triangular attention mask, used to ensure that the current position only focuses on itself and previous historical tokens. Softmax is a mathematical function that maps a vector to a probability distribution, ensuring that all probabilities are greater than 0 and the sum of probabilities equals 1. Attention represents the attention mechanism, which is used to calculate the correlation between each token and other tokens. Mask is the calculation process to ensure that the current position only focuses on itself and previous historical tokens. T represents the vector transpose operation.

7. A research and development system based on an auxiliary large model and a research and development auxiliary framework according to claim 6, characterized in that, The self-attention head calculation: ； Based on the above calculation formula, a total of h self-attention head vectors are calculated; V represents the value matrix, which is the "attention score matrix Attention(l)" used for the "dot product" calculation when calculating the "self-attention head vector head"; l represents the number of the "decoding block computation layer" from layer 1 to layer 80. Since the "multi-head self-attention mechanism" is a sub-computation process in the "decoding block computation layer", each l-layer "decoding block computation layer" will also contain a "multi-head self-attention mechanism" calculation process; in the "self-attention head calculation" process, the input sequence is multiplied by the query matrix Q, the key matrix K, and the value matrix V respectively, and then they are concatenated according to certain rules. Finally, a linear transformation is performed to obtain the vector result value of the output sequence. Output results calculated by the multi-head self-attention mechanism ； Where Concat is the horizontal concatenation calculation of h self-attention head vectors; W o （l） This is the trainable weight parameter matrix for multi-head self-attention vectors when calculating the output of nonlinear projection transformation. In the "multi-head self-attention mechanism", h self-attention head vectors can be calculated in parallel, and this parallel calculation process can significantly enhance the efficiency of code-assisted training and inference of large models.

8. A research and development system based on an auxiliary large model and a research and development auxiliary framework according to claim 7, characterized in that, Each "decoding block computation layer" performs two calculations, which include: (1) Residual connections after multi-head self-attention mechanism computation; ； Among them, H （l） The input hidden state vector of the l-th decoder block is calculated; AttOut （l） This is the output result calculated by the multi-head self-attention mechanism; (2) Residual connections after feedforward neural network computation; ； Among them, FFNOut （l） The output result calculated by the feedforward neural network; H （l+1） Calculate the output hidden state vector of the layer for the l-th decoding block; (3) Normalized calculation: The intermediate process "layer normalization" targets the calculation result Z of the "residual connectivity after multi-head self-attention mechanism". （l） norm The normalization calculation is performed using the following formula: ； When l is 1 to 80 layers, Z （1） The result vector calculated by the "residual connection" model layer in the computation layer of the l-th decoding block; H （l） norm When l is from 1 to 80, Z(l) is the normalized result vector output by the "residual connection" model layer in the l-th decoding block computation layer after "layer normalization" calculation; Mean is the average value of all input vectors in the layer normalization calculation formula, and "l" represents the l-th decoding block computation layer.

9. A research and development system based on an auxiliary large model and a research and development auxiliary framework according to claim 8, characterized in that, The feedforward neural network module is the core nonlinear projection transformation calculation process of each "decoding block calculation layer". Its input is the normalized calculation result of the intermediate process layer, and its calculation formula is as follows: ； SwiGLU is a non-linear activation function used to help the code assist large models in performing non-linear fitting and learning of training data, which can significantly enhance the expressive power of the model; W(l)down, W(l)up, and W(l)gate are weight vector matrices required for performing "vector matrix multiplication" in the "feedforward neural network" model layer in the l-th decoding block computation layer. This represents "dot multiplication"; Swiglu is a non-linear activation function that combines the ideas of the Swish function and gated linear units.

10. A research and development system based on an auxiliary large model and a research and development auxiliary framework according to claim 9, characterized in that, The R&D support framework system includes: The code completion and generation module completes and generates development code that meets specific requirements. The code knowledge base module stores, retrieves, and consolidates R&D knowledge assets. The code explanation module generates code explanations, reviews, and optimization plans. The code comment module generates compliant and maintainable code comments; Generate unit test modules to generate high-coverage compliant test cases; The code Q&A module answers technical and business logic questions; The code completion and generation module, code knowledge base module, code explanation module, code comment module, unit test generation module, and code Q&A module are directly interconnected in both directions.

Images