Coding activity task (CAT) evaluation for source code generator
By separating the code into constituent blocks and generating equivalent blocks, calculating the decoding score, combining offline and online evaluation, the problem of coarse-grained code generators in the prior art is solved, and a human-centered fine-grained evaluation is achieved, reducing the Evaluate costs and improve the quality of the code generator.
Patent Information
- Application Number
- CN202380035253.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-21
- Filing Date
- 2023-03-24
- Publication Date
- 2024-11-29
AI Technical Summary
Existing code generator evaluation methods rely on the similarity between code and benchmark code, and cannot effectively capture functional equivalence, correctness, and complexity, resulting in coarse-grained evaluation, making it difficult to distinguish the differences between almost correct and large number of errors, and Relying on functional testing, time-consuming and expensive.
By separating the software code into constituent blocks, generating equivalent blocks with code generators, calculating decoding scores, and aggregated into code generator scores, providing finer-grained evaluations, combining offline and online evaluations, introducing human-centric code Utility measurement, using syntax parsing and semantic similarity engines, implements fine-grained evaluation of code generators.
It provides a more human-centered evaluation method, which can more meticulously allocate credit, capture task complexity and human cognitive requirements, reduce evaluation costs, and quickly select the best code generator and generate better output code.
Smart Images

Figure CN119053946A_ABST
Abstract
Description
Background Art
[0001] Advances in artificial intelligence (AI), particularly in autoregressive large language models that use deep learning to produce human-like text, have enabled new applications (e.g., code generators) that generate software source code and improve productivity for developers. For example, a code generation model (code generator) that has been trained using a large language model architecture is able to take a description of a function (and any additional context, e.g., surrounding functions or files) and write multiple lines of code to execute the function. Unfortunately, evaluating code generator systems is challenging due to the lack of validated metrics and the cost of labeled data.
[0002] Current evaluation schemes rely on similarity between the generated code and a ground truth (e.g., previously existing code written by humans), and do not capture the concepts of functional equivalence, correctness, or complexity (e.g., cognitive difficulty or computational complexity), and are therefore not human-centric metrics. Current evaluation schemes that rely on pass / fail functional testing require execution (with security implications) and have too coarse granularity (e.g., complete functionality), and therefore do not provide the insight needed to distinguish between almost correct results (e.g., only one line out of dozens is wrong) and a large number of errors. Other current evaluation schemes that measure the acceptance or survival rate of generated functions are more human-centric, but also fail to capture the concepts of functional equivalence, correctness, or complexity. Therefore, both selecting well-performing code generators and improving machine learning (ML) training of a pool of code generators remain time-consuming and expensive. Summary of the invention
[0003] The following examples of the present disclosure are described in detail with reference to the accompanying drawings listed below. The following summary is provided to illustrate some examples disclosed herein. However, this does not mean that all examples are limited to any specific configuration or operation sequence.
[0004] A solution for evaluating source code generators includes: separating, by a processor, each of a plurality of input segments of software code into a plurality of constituent blocks; generating, with each of a first plurality of code generators, an equivalent block for at least one constituent block in each of the plurality of constituent blocks, thereby producing a plurality of equivalent blocks corresponding to each code generator; determining, for each code generator, a first decoding score for each of the plurality of equivalent blocks corresponding to the code generator; aggregating, for each code generator, the first decoding score across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting, from the first plurality of code generators, a second plurality of code generators having a smaller count than the first plurality of code generators based at least on a ranking of the first aggregate scores for the first plurality of code generators; and generating, with each of the second plurality of code generators, an output segment of the software code. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Examples of the present disclosure are described in detail below with reference to the accompanying drawings listed below:
[0006] Figure 1 An example apparatus is shown that advantageously provides Coding Activity Task (CAT) evaluation for a source code generator;
[0007] Figure 2 As shown in Figure 1 Separating an input segment of software code into a plurality of constituent blocks and generating equivalent blocks occurs in an example of an apparatus;
[0008] Figure 3A and Figure 3B Shown in Figure 1 Examples of CAT Replacement and Test Evaluation (CAT-RT) used in some examples of devices;
[0009] Figure 4 Shown in Figure 1 Examples of CAT Similarity Assessment (CAT-S) used in some examples of the apparatus;
[0010] Figure 5 Shown in Figure 1 Examples of CAT-Online Performance Evaluation (CAT-OP) used in some examples of devices;
[0011] Figure 6 shows how it can be Figure 1 Some examples of devices that occur together with a process flow of an example CAT assessment using CAT-RT, CAT-S, and CAT-OP;
[0012] Figure 7 Shows the description and Figure 1A flowchart of exemplary operations associated with an apparatus;
[0013] Figure 8 Shows the description and Figure 1 Another flowchart of exemplary operations associated with an apparatus of;
[0014] Fig. 9 is a block diagram of an example computing environment suitable for implementing some of the various examples disclosed herein.
[0015] Corresponding reference numerals indicate corresponding parts throughout the drawings. DETAILED DESCRIPTION
[0016] Various examples will be described in detail with reference to the accompanying drawings. Where preferred, the same reference numerals will be used throughout the drawings to refer to the same or similar components. References to specific examples and implementations made throughout this disclosure are provided for illustrative purposes only and are not intended to limit all examples unless otherwise indicated.
[0017] The solution for evaluating source code generators uses offline and online evaluation stages.Offline evaluation includes separating each input segment of multiple input segments of software code into multiple constituent blocks.Each code generator (of multiple code generators) generates an equivalent block corresponding to each constituent block.(For each code generator) a decoding score is determined for each equivalent block, and the decoding score is aggregated across equivalent blocks to provide an aggregate score for each code generator.In some examples, the ranking of the aggregate score is used to screen a smaller number of code generators for online evaluation.For this stage, the code generator outputs a segment of software code, and the user acceptance of the output of the code generator can be used for further ranking and screening.Some examples weight the decoding score according to the code utility evaluation (which can be based on a learned human-centric code measurement operationalized by such as complexity, readability, error probability, etc.) of the constituent blocks for which equivalent blocks are generated.
[0018] Aspects of the present disclosure provide a more human-centric evaluation of code generators by scoring code generator performance in constituent blocks (rather than entire functions) and aggregating scores (based on the human-centric measures mentioned above). In some examples, online evaluation (i.e., determining acceptance rates) is used in tandem with offline evaluation, although online evaluation is limited to code generators that achieve superior scores in offline testing.
[0019] The solution provided goes beyond functional scope, assigns credit in a more fine-grained manner, and encodes human / task attributes and online metrics that take into account task complexity and human cognitive requirements. The Coding Active Task (CAT) approach represents a human-centered evaluation of code generators. In some examples, CAT scores are derived as follows: (1) a target task (function or class) is parsed into small extracted chunks using a syntax parser with corresponding task labels; (2) completion is obtained for each chunk (equivalent chunk); (3) a score is derived for each task in one or both of two ways: (a) the extracted building blocks are replaced with equivalent chunks in the target function and scores are calculated based on unit test results, and (b) the equivalent chunks are compared to the extracted building blocks using a semantic similarity engine to calculate scores based on similarity.
[0020] The first scoring option operates on constituent blocks rather than entire discovered functions. This allows test failures to be attributed to equivalent blocks (since presumably unmodified functions may pass). The second scoring option determines no code that is dependent on the availability of unit tests or execution. The final offline CAT score is the (in some examples weighted) average of the CAT scores for all small-scale blocks.
[0021] This offline CAT score can then be used to assess the quality of each code generator, allowing a smaller number of code generators to be prioritized for online evaluation. This approach introduces a cost-effective label-free solution for fine-grained human-centric evaluation of code generators with flexible granularity. Examples use weights based on code utility and provide a low-efficiency path to scale offline evaluation to multiple languages.
[0022] Figure 1 An example apparatus is shown that advantageously provides CAT evaluation for source code generators. In some examples, apparatus 100 uses Fig. 9 The first plurality of code generators 134 in the code generator pool 130 are implemented in one or more examples of the computing device 900 of the apparatus 100. In the apparatus 100, the first plurality of code generators 134 in the code generator pool 130 are evaluated by the offline evaluation 110, which utilizes the baseline (e.g., human-written) source code from the projects in the source code repository 140 (e.g., from the project 141 and the project 142). Based on the performance, the first plurality of code generators 134 are screened into a smaller second plurality of code generators 135, which are passed to the online evaluation 120.
[0023] The best performing code generator 131 is identified by the online evaluation 120 and used to generate new output segments of software code 153 that support the transcoding project. This process is much faster than a user trying multiple code generators to manually select the one with the best performance, and produces superior results (e.g., more superior output segments of software code 153) than selection methods that do not use human-centric evaluation.
[0024] As shown, first plurality of code generators 134 includes code generator 131, code generator 132, and code generator 133. Although only three code generators are shown, it should be understood that some examples may have tens or hundreds (or more) of code generators in first plurality of code generators 134. Second plurality of code generators 135 includes only code generator 131 and code generator 132, but some examples may include a larger number.
[0025] Project 141 has a segment of software code 151, test 161a, and test 161b. Project 142 has a segment of software code 152, test 162a, and test 162b. Tests 161a to 162b may be unit tests. It should be understood that there may be different numbers of tests in different projects.
[0026] The offline evaluation 110 uses a CAT Replacement and Testing Strategy (CAT-RT) 300, which replaces constituent blocks of a segment of software code 151 and / or 152 with equivalent blocks and calculates a score based on the test results. Further details of the CAT-RT 300 are described in detail in the Figure 2 , Figure 3A and Figure 3B The offline evaluation 110 also uses a CAT similarity strategy (CAT-S) 400 to compare equivalent blocks to constituent blocks using a semantic similarity engine 401. Further details of the CAT-S 400 are provided in the context of Figure 2 and Figure 4 is provided in the content.
[0027] The CAT-RT / S aggregator 111 aggregates the code generator scores across equivalent blocks of different software code segments into a first aggregate score 112 for each code generator. Additional details are provided in the Figure 6In some examples, the ranking engine 113 optionally ranks the first plurality of code generators 134 by their first aggregate scores 112 as a ranking 114 (ranked list), and the selection logic 115 selects the top N (e.g., in some examples, N=2) code generators to pass to the online evaluation 120 as the second plurality of code generators 135. That is, the second plurality of code generators 135 are a filtered version of the first plurality of code generators 134.
[0028] The online evaluation 120 uses the CAT Online Performance Policy (CAT-OP) 500, further details of which are provided in the Figure 5 CAT-OP 500 generates an acceptance score, which can be aggregated (in some examples) with offline results (e.g., first aggregate score 112) by offline / online aggregator 121 into a second aggregate score 122. For further details on acceptance scores, see Figure 5 The ranking engine 123 ranks the second plurality of code generators 135 by their second aggregated scores 122 into a ranking 124 (ranked list) so that the best code generator (eg, in the illustrated example, code generator 131 ) can be identified.
[0029] In some examples, the CAT OP labels and scores for a decoding session are applied as a vector representation of the decoding session and used for downstream tasks such as performance monitoring and anomaly detection. The vector representation can be implemented as follows: Given N different CAT block types, a session is represented as a vector of size N, where each position corresponds to the count of each block type in the session.
[0030] The code generators in the code generator pool 130 are trained by a machine learning (ML) trainer 160 using training data 162. Because training data 162 is very expensive, feedback from CAT evaluation, in offline evaluation 110 and online evaluation 120 is utilized by the trainer 160 to further train the code generators in the code generator pool 130. This is achieved through the finer granularity provided by the offline evaluation 110 solution of the present disclosure, which enables visibility into the types of building blocks where the code generator performance is weakest.
[0031] Figure 2 An alternative process 200 is shown which includes separating an input segment of software code 151 into a plurality of constituent blocks 606 (see further details in Figure 6) and the generation of equivalent blocks 212. The input segment of the software code 151 is passed to a block extractor 208, which decomposes a large function into multiple blocks, for example, by using a syntax parser. In the example shown, the input segment of the software code 151 is decomposed into five building blocks: building block 201, building block 202, building block 203, building block 204, and building block 205. Task label 252 is created for building block 202, and task label 254 is created for building block 204 (task labels can also be created for other building blocks).
[0032] Additionally, in some examples, code utility is determined at least for a constituent block for which a corresponding equivalent block is generated by a code generator. Thus, code utility 253 is determined for constituent block 202, and code utility label 255 is determined for constituent block 204. Code utilities 253 and 255 are used to determine weights to be applied when aggregating code generator scores (e.g., Figure 6 The weights 630 are weighted so that more complex blocks are weighted in the final score. Weighting options include weighting higher complexity more heavily, weighting by frequency of occurrence in the task distribution, and / or weighting according to the expected cost of an error in the corresponding portion of the code (i.e., criticality).
[0033] Code utility is not just lines of code, but can include additional proxies for complexity, such as the number of reported failures per line, the number of changes requested per line, and other metrics. Such other metrics can include the frequency with which tasks are associated with stack traces and errors, build failures, readability, naturalness, and other metrics. While assignment statements and arithmetic operations can be considered simple, error handling instructions can be considered to have higher complexity.
[0034] In the illustrated example, at least by the code generator 131, the constituent blocks 202 and 204 are selected for generating the equivalent blocks 212 and 214, respectively. That is, the code generator 131 is instructed to write (generate) the equivalent block 212, which performs the same tasks as the constituent block 202 and can therefore replace the constituent block 202. In addition, the code generator 131 is instructed to write (generate) the equivalent block 214, which performs the same tasks as the constituent block 204 and can therefore replace the constituent block 204. Different examples may select different numbers of constituent blocks for which corresponding equivalent blocks are generated.
[0035] The equivalent block 212 replaces the building block 202 in the input segment of the software code 151 to produce a modified segment 251 that is tested in the CAT-RT 300, as described with respect to FIG. Figure 3A and Figure 3BThe equivalent block 214 replaces the building block 204 in the input segment of the software code 151 to produce a modified segment 251a that is also tested in the CAT-RT 300, as described with respect to Figure 3A and Figure 3B described.
[0036] Substitution process 200 is performed for each code generator in first plurality of code generators 134 such that the role shown for code generator 131 is also performed by code generators 132 and 133, each producing their own set of modified segments.
[0037] Figure 3A An example of a CAT-RT 300 is shown in which a modified segment 251 (generated by a replacement process 200) is subjected to a single test 161a by a test manager 301. This provides a single test result 302, which in some examples is a binary pass / fail. However, some examples may produce a scalar test result 302 or a multi-valued test result 302. The CAT-RT 300 is executed for each combination of a code generator and a modified segment used in the offline evaluation 110.
[0038] Figure 3B An example of a CAT-RT 300a is shown in which a modified segment 251 is subjected to a plurality of tests 361, specifically test 161a and test 161b, by a test manager 301. This provides a plurality of test results 360, specifically test result 360a for test 161a and test result 360b for test 161b. In some examples, test 161a and test 161b test different blocks of the modified segment 251. An aggregator 362 aggregates the test results 360a and 360b into a test result 302. Although CAT-RT 300a is described as using multiple tests and CAT-RT 300 is described as using only a single test, the offline assessment 110 may generally use CAT-RT 300a interchangeably with CAT-RT 300.
[0039] Figure 4The example of CAT-S 400 is shown. Semantic similarity engine 401 is for semantic similarity comparison building block 202 and equivalent block 212, and distributes semantic similarity 402 (for example, similarity score). The example of semantic similarity measurement includes but is not limited to edit distance, neural similarity through learning, bilingual evaluation replacement (BLEU), software similarity measurement (MOSS) and Gisting evaluation (recall-oriented understudy for Gisting evaluation, ROUGE) for recall replacement. Based on the specific original metric for semantic similarity, lower or higher value indicates superior performance. Normalization causes different original metrics to become consistent scoring themes. Semantic similarity 402 is determined for each combination of code generator and the modified segment used in offline evaluation 110.
[0040] Figure 5 An example of a CAT-OP 500 executed for each of the second plurality of code generators 135 (specifically, code generator 131 and code generator 132) is shown. In some examples of the apparatus 100, the CAT-OP 500 is optional. Code generator 131 is used to generate new output segments of software code 511 and new output segments of software code 512. Acceptance information 501 is determined based on whether the output segments of software codes 511 and 512 are used or discarded by users working on the software project. Acceptance score 502 uses acceptance information 501 to determine an acceptance score 503 for code generator 131.
[0041] Similarly, the code generator 132 is used to generate new output segments of software code 521 and new output segments of software code 522. Acceptance information 501a is determined based at least in part on how software developers process new output segments of software code 521 and 522 in other projects. For example, segments can be submitted, saved (unsubmitted), used as is, used after editing, or discarded completely. Other metrics include time to be accepted, percentage of code written by the code generator, percentage of code edited after a certain amount of time, and survival rate of unedited. In some examples, CAT OP tags and scores for decoding sessions are applied as vector representations of decoding sessions and used for downstream tasks such as performance monitoring and anomaly detection. The vector representation can be implemented as follows: given N different CAT block types, the session is represented as a vector of size N, where each position corresponds to a count of each block type in the session. Acceptance score 502 uses acceptance information 501a to determine acceptance score 503a for code generator 132.
[0042] Figure 6600 is shown as a process flow that may occur with some examples of the apparatus 100. The process flow 600 will Figures 1 to 5 The various aspects of the software code 601 are linked together. The multiple input segments of the software code 601 (including the segments of the software code 151 and the software code 152) are each separated into multiple building blocks. For example, the segment of the software code 151 is separated into multiple building blocks 606, and the building blocks 606 include at least the building blocks 202 and 204 (see also Figure 2 ), and the segments of the software code 152 are separated into a plurality of constituent blocks 608, which at least include constituent blocks 602 and constituent blocks 604.
[0043] At least one of each of the plurality of building blocks 606 and 608 is provided to each of the first plurality of code generators 134 (which includes code generators 131 to 133). This generates a plurality of equivalent blocks for each code generator. For example, code generator 131 generates an equivalent block 212 for building block 202 and an equivalent block 621 for building block 602; code generator 132 generates an equivalent block 612 for building block 202 and an equivalent block 622 for building block 602; and code generator 133 generates an equivalent block 613 for building block 202 and an equivalent block 623 for building block 602.
[0044] In addition, the code generator 131 generates an equivalent block 214 for the constituent block 204. The code generators 132 and 133 also generate corresponding equivalent blocks for the constituent block 204, although these are not shown for clarity of illustration.
[0045] Thus, there are multiple equivalent blocks for each code generator in the first plurality of code generators 134. For example, code generator 131 generates multiple equivalent blocks 626, which include equivalent blocks 212, 214, and 621; code generator 132 generates multiple equivalent blocks 627, which include equivalent blocks 612 and 622; and code generator 133 generates multiple equivalent blocks 628, which include equivalent blocks 613 and 623. As described above, this set of equivalent blocks is used by offline evaluation 110.
[0046] The set of first decoding scores includes first decoding scores 636, first decoding scores 637, and first decoding scores 638 (corresponding to the respective code generators in code generators 131, 132, and 133), which are weighted by weight 630 according to code utilities 253 and 254 (derived from other factors such as task complexity). These weighted first scores are aggregated by CAT-RT / S aggregator 111 into first aggregated scores 112, which are used to screen first plurality of code generators 134 into second plurality of code generators 135.
[0047] In online evaluation 120, CAT-OP 500 generates acceptance scores 503 for each of code generators 131 and 132 (eg, acceptance score 503a for code generator 132 is Figure 6 denoted as acceptance score 503a). Figure 6 Aggregation of the acceptance score 503 and the first aggregated score 112 into a second aggregated score 122 by the offline / online aggregator 121 is shown.
[0048] Figure 7 A flowchart 700 illustrating exemplary operations associated with an example of apparatus 100 is shown. In some examples, the operations described with respect to flowchart 700 are performed by Fig. 9 Executed by the computing device 900 of FIG. 1 . In operation 702 , the flowchart 700 begins by retrieving the input segments of the software codes 151 and 152 and the tests 161 a to 162 b from the source code repository 140 .
[0049] Operation 704 uses processor 914 (see Fig. 9 ) separates each of the plurality of input segments of the software code 601 into a plurality of building blocks 606 and 608. Operation 706 generates corresponding task tags (e.g., task tags 252 and 254) for at least some of the building blocks (for at least each of the building blocks for which equivalent blocks are generated). Operation 708 determines code utility (e.g., complexity 253 and 253) for at least some of the building blocks (for at least each of the building blocks for which equivalent blocks are generated), and assigns weights (e.g., weights 630) based at least on the determined complexity.
[0050] For each code generator in the first plurality of code generators 134, operations 710 and 712 are performed for each constituent block (for which an equivalent block is generated). That is, operation 710 is the beginning of a double nested loop. Operation 710 includes: for at least one constituent block in each plurality of constituent blocks 606 and 608, generating an equivalent block using each code generator in the first plurality of code generators 134, thereby generating a plurality of equivalent blocks corresponding to each code generator.
[0051] Operation 712 determines, for each code generator (131 to 133), a first decoding score (636, 637, or 638) for each of the plurality of equivalent blocks corresponding to the code generator. This is performed using operations 714 to 722. Operation 714 replaces a constituent block with an equivalent block in an input segment of the software code from which the equivalent block is generated, thereby generating a modified segment. For example, in an input segment of the software code 151 from which the equivalent block 212 is generated, the constituent block 202 is replaced with the equivalent block 212, thereby generating a modified segment 251.
[0052] Operation 716 determines the test result 302 of testing the modified segment 251. In some examples, operation 716 includes operation 718, which performs multiple tests 361 on the modified segment 251 and aggregates the test results 360a and 360b of the multiple tests 361 on the modified segment 251. In such an example, the test result 302 includes an aggregation of the test results 360a and 360b of the multiple tests 361. Operation 720 determines the semantic similarity 402 between the constituent block 202 and the equivalent block 212, and operation 722 aggregates the test result 302 and the semantic similarity 402 into a first decoding score 636 for the code generator 131. For code generator 132 , test result 302 and semantic similarity 402 (for code generator 132 ) are aggregated into a first decoding score 637 , and for code generator 133 , test result 302 and semantic similarity 402 (for code generator 133 ) are aggregated into a first decoding score 638 .
[0053] For the current code generator, decision operation 724 determines whether there are additional building blocks to be replaced with equivalent blocks. If so, the flowchart 700 returns to operation 710. In the next pass operation 710 to 722, additional equivalent blocks are generated for at least one additional building block (e.g., for the equivalent block 214 of the building block 204). For each code generator, a second decoding score is determined for the additional equivalent block. Otherwise, operation 726 aggregates the first decoding score 636, 637, or 638 for each code generator across multiple equivalent blocks corresponding to the code generator into a first aggregate score 112 for the code generator. In some examples, operation 726 also includes: for each code generator, aggregating the second decoding score into a first aggregate score for the code generator across multiple additional equivalent blocks corresponding to the code generator. In some examples, as part of operation 726, operation 728 weights each first decoding score (and second decoding score) according to the code utility of its corresponding building block.
[0054] Decision operation 730 determines whether additional code generators are to be evaluated. If so, flowchart 700 returns to operation 710. Otherwise, operation 732 ranks the first plurality of code generators 134 based on at least the first aggregated score 112. Operation 734 selects (screens) the second plurality of code generators 135 from the first plurality of code generators 134 based on at least the ranking 114 of the first aggregated score 112. Operations 702 to 734 together comprise an offline evaluation process.
[0055] Operation 736 generates output segments of software code (e.g., output segments of software code 511 and 521 for code generators 131 and 132, respectively) with each code generator in the second plurality of code generators 135. Operation 738 determines acceptance information 501 for each output segment in the software code, and also determines, for each code generator in the second plurality of code generators 135, an acceptance score 503 based at least on the acceptance information 501 for the output segment of its software code.
[0056] For each code generator in the second plurality of code generators 135, operation 740 aggregates the first aggregate score 112 with the acceptance score 503 into a second aggregate score 122. Operation 742 ranks the second plurality of code generators 135 based on at least the second aggregate score 122. Operation 744 includes selecting a best code generator (e.g., code generator 131) from the second plurality of code generators 135 based on at least the ranking 124 of the second aggregate score 122 of the second plurality of code generators 135.
[0057] The code generators are used to produce source code by generating output segments of software code 153 in operation 746. Operation 748 feeds the first aggregate scores 112 for the various code generators back to the trainer 160, and also feeds the second aggregate scores 122 back to the trainer 160. The trainer then further trains the selected set of code generators.
[0058] Figure 8 A flowchart 800 illustrating exemplary operations associated with an example of apparatus 100 is shown. In some examples, the operations described with respect to flowchart 800 are performed by Fig. 9 The flowchart 800 starts with operation 802, which includes separating, by a processor, each of a plurality of input segments of a software code into a plurality of constituent blocks.
[0059] Operation 804 includes: for at least one constituent block in each of the plurality of constituent blocks, generating an equivalent block using each of the first plurality of code generators, thereby generating a plurality of equivalent blocks corresponding to each code generator. Operation 806 includes: for each code generator, determining a first decoding score for each of the plurality of equivalent blocks corresponding to the code generator. Operation 808 includes: for each code generator, aggregating the first decoding score across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator.
[0060] Operation 810 includes selecting a second plurality of code generators from the first plurality of code generators based at least on a ranking of first aggregate scores for the first plurality of code generators, the second plurality of code generators having a smaller count than the first plurality of code generators. Operation 812 includes generating output segments of software code using each of the second plurality of code generators. Additional Examples
[0061] An example system includes: a processor; and a computer storage medium storing instructions that, when executed by the processor, are operable to: separate, by the processor, each of a plurality of input segments of software code into a plurality of constituent blocks; for at least one constituent block in each of the plurality of constituent blocks, generate an equivalent block using each of a first plurality of code generators, thereby producing a plurality of equivalent blocks corresponding to each code generator; for each code generator, determine a first decoding score for each of the plurality of equivalent blocks corresponding to the code generator; for each code generator, aggregate the first decoding score across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; select a second plurality of code generators from the first plurality of code generators based at least on a ranking of the first aggregate scores for the first plurality of code generators, the second plurality of code generators having a smaller count than the first plurality of code generators; and generate an output segment of the software code using each of the second plurality of code generators.
[0062] An example method includes: separating, by a processor, each of a plurality of input segments of a software code into a plurality of constituent blocks; generating, for at least one constituent block in each of the plurality of constituent blocks, an equivalent block using each of a first plurality of code generators, thereby producing a plurality of equivalent blocks corresponding to each code generator; determining, for each code generator, a first decoding score for each of the plurality of equivalent blocks corresponding to the code generator; aggregating, for each code generator, the first decoding score across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting, from the first plurality of code generators, a second plurality of code generators having a smaller count than the first plurality of code generators based at least on a ranking of the first aggregate scores for the first plurality of code generators; and generating, using each of the second plurality of code generators, an output segment of the software code.
[0063] One or more example computer storage devices having computer executable instructions stored thereon, which when executed by a computer, cause the computer to perform operations, the operations comprising: separating, by a processor, each of a plurality of input segments of a software code into a plurality of constituent blocks; generating, for at least one constituent block in each of the plurality of constituent blocks, an equivalent block using each of a first plurality of code generators, thereby producing a plurality of equivalent blocks corresponding to each code generator; for each code generator, determining a first decoding score for each of the plurality of equivalent blocks corresponding to the code generator; for each code generator, aggregating the first decoding score across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting, from the first plurality of code generators, a second plurality of code generators having a smaller count than the first plurality of code generators based at least on a ranking of the first aggregate scores for the first plurality of code generators; and generating, using each of the second plurality of code generators, an output segment of the software code.
[0064] Alternatively or in addition to other examples described herein, examples include any combination of: - for each output section of the software code, determining acceptance information; - for each code generator of the second plurality of code generators, determining an acceptance score based at least on acceptance information for an output segment of its software code; - determining said first decoding score comprises: replacing a constituent block with the equivalent block in an input segment of the software code from which the equivalent block is generated, thereby producing a modified segment; - determining said first decoding score comprises determining a test result of testing the modified segment; -determining the test results includes performing a plurality of tests on the modified segment; - the test result comprises an aggregation of test results of a plurality of tests; - determining the first decoding score further comprises determining a semantic similarity between the constituent block and an equivalent block; - determining a first decoding score further comprises aggregating the test results and the semantic similarity into the first decoding score; - aggregating the first decoding scores into a first aggregate score comprises weighting each first decoding score according to the code utility of its corresponding constituent block; - for at least one additional building block in each of the plurality of building blocks, generating an additional equivalent block using each code generator in the first plurality of code generators, thereby generating a plurality of additional equivalent blocks corresponding to each code generator; - for each code generator, determining a second decoding score for each additional equivalent block in the plurality of equivalent blocks corresponding to the code generator; - for each code generator, aggregating the second decoding scores into a first aggregate score for the code generator across a plurality of additional equivalent blocks corresponding to the code generator; -Generate corresponding task labels for each building block; - Determine the code utility of each constituent block; - ranking the first plurality of code generators based at least on the first aggregate score; - for each code generator of the second plurality of code generators, aggregating the first aggregate score with the acceptance score into a second aggregate score; - ranking the second plurality of code generators based at least on the second aggregate score; and - selecting a best code generator from the second plurality of code generators based at least on the ranking of the second aggregated scores for the second plurality of code generators; and generating an output segment of the software code using the best code generator.
[0065] Although aspects of the disclosure have been described in terms of various examples with their associated operations, those skilled in the art will appreciate that combinations of operations from any number of the different examples are also within the scope of aspects of the disclosure. Example operating environment
[0066] Fig. 9 900 for implementing various aspects disclosed herein, and is generally designated as computing device 900. In some examples, one or more computing devices 900 are provided for on-premises computing solutions. In some examples, one or more computing devices 900 are configured as cloud computing solutions. In some examples, a combination of local and cloud computing solutions are used. Computing device 900 is only one example of a suitable computing environment and is not intended to impose any limitation on the scope of use or functionality of the examples disclosed herein, whether used alone or as part of a larger collection.
[0067] Computing device 900 should not be interpreted as having any dependency or requirement related to any one or combination of the components / modules shown. The examples disclosed herein can be described in the general context of computer code or machine-usable instructions, including computer executable instructions executed by a computer or other machine (such as a personal data assistant or other handheld device), such as program components. Generally, program components including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. The disclosed examples can be put into practice in various system configurations, including personal computers, laptop computers, smart phones, mobile tablet computers, handheld devices, consumer electronics, professional computing devices, etc. When tasks are performed by remote processing devices linked by a communication network, examples of the present disclosure can also be put into practice in a distributed computing environment.
[0068] The computing device 900 includes a bus 910 that directly or indirectly couples the following devices: memory 912, one or more processors 914, one or more presentation components 916, input / output (I / O) ports 918, I / O components 920, power supply 922, and network components 924. Although the computing device 900 is depicted as appearing to be a single device, multiple computing devices 900 can work together and share the depicted device resources. For example, the memory 912 can be distributed across multiple devices, and the processor 914 can be housed with different devices.
[0069] Bus 910 represents what may be one or more buses (such as an address bus, a data bus, or a combination thereof). Although shown with lines for clarity, Fig. 9 , but alternative representations may be used to depict various components. For example, in some examples, presentation components such as display devices are I / O components, and some examples of processors have their own memory. There is no distinction between categories such as "workstation", "server", "laptop", "handheld device", etc., because all of these are in Fig. 9 912 is within the scope of and within the scope of references to "computing device" herein. The memory 912 may take the form of computer storage media referenced below and is operable to provide storage of computer readable instructions, data structures, program modules, and other data for the computing device 900. In some examples, the memory 912 stores one or more of an operating system, a general-purpose application platform, or other program modules and program data. The memory 912 is thus capable of storing and accessing data 940 and instructions 942 that can be executed by the processor 914 and configured to perform various operations of the present disclosure.
[0070] In some examples, memory 912 includes computer storage media. Memory 912 may include any number of memories associated with or accessible by computing device 900. Memory 912 may be internal to computing device 900 (e.g., Fig. 9 ), external to computing device 900 (not shown), or both (not shown). Additionally or alternatively, memory 912 may be distributed across multiple computing devices 900, such as in a virtualized environment where instruction processing is performed on multiple computing devices 900. For purposes of this disclosure, "computer storage media," "computer storage memory," "memory," "computer storage device," and "memory device" are synonymous terms for memory 912, and none of these terms include carrier waves or propagated signaling.
[0071] (Multiple) processors 914 can include any number of processing units that read data from various entities (such as memory 912 or I / O components 920). Specifically, (multiple) processors 914 are programmed to execute computer executable instructions for implementing various aspects of the present disclosure. Instructions can be executed by a processor, by multiple processors within the computing device 900, or by a processor outside the client computing device 900. In some examples, (multiple) processors 914 are programmed to execute instructions, such as those shown in the flowcharts discussed below and depicted in the accompanying drawings. In addition, in some examples, (multiple) processors 914 represent the implementation of simulation technology for performing the operations described herein. For example, the operations can be performed by a simulated client computing device 900 and / or a digital client computing device 900. (Multiple) presentation components 916 present data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc. Those skilled in the art will understand and appreciate that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through a speaker, wirelessly between computing devices 900, across a wired connection, or otherwise. I / O ports 918 allow computing device 900 to be logically coupled to other devices including I / O components 920, some of which may be built-in. Example I / O components 920 include, for example, but are not limited to, a microphone, a joystick, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like.
[0072] The computing device 900 can operate in a networked environment using logical connections to one or more remote computers via a network component 924. In some examples, the network component 924 includes a network interface card and / or computer executable instructions (e.g., a driver) for operating a network interface card. Communications between the computing device 900 and other devices can occur over any wired or wireless connection using any protocol or mechanism. In some examples, the network component 924 can be operable to communicate between devices using short-range communication technologies (e.g., near field communication (NFC), Bluetooth, etc.) using a public, private, or hybrid (public and private) transport protocol. TM Brand communication, etc.), or a combination thereof. Network component 924 communicates with remote resources 928 (e.g., cloud resources) across network 930 via wireless communication links 926 and / or wired communication links 926a. Various different examples of communication links 926 and 926a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed through the Internet.
[0073] Although described in conjunction with the example computing device 900, the examples of the present disclosure can be implemented with many other general or special computing system environments, configurations or devices. Examples of well-known computing systems, environments and / or configurations applicable to various aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices in wearable or accessory form factors (e.g., watches, glasses, headphones or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, etc. Such systems or devices can accept input from a user in any manner, including from an input device such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and / or via voice input.
[0074] Examples of the present disclosure may be described in the general context of computer executable instructions (such as program modules) executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. Computer executable instructions may be organized into one or more computer executable components or modules. Typically, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Aspects of the present disclosure may be implemented with such components or modules of any number and organization. For example, aspects of the present disclosure are not limited to specific computer executable instructions or specific components or modules shown in the accompanying drawings and described herein. Other examples of the present disclosure may include different computer executable instructions or components with more or less functions than those shown and described herein. In examples involving general-purpose computers, aspects of the present disclosure transform general-purpose computers into special-purpose computing devices when configured to execute instructions described herein.
[0075] As an example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable memory implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive for communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagation signals. Computer storage media for the purposes of this disclosure are not signals themselves. Exemplary computer storage media include hard disks, flash drives, solid-state memories, phase change random access memories (PRAM), static random access memories (SRAM), dynamic random access memories (DRAM), other types of random access memories (RAM), read-only memories (ROM), electrically erasable programmable read-only memories (EEPROM), flash memory or other memory technologies, compact disk read-only memories (CD-ROM), digital versatile disks (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by computing devices. In contrast, communication media typically embodies computer readable instructions, data structures, program modules, etc. in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
[0076] The execution or execution order of the operations in the examples of the present disclosure shown and described herein is not necessary, and can be executed in different sequential ways in various examples.For example, it is contemplated that before, simultaneously or after another operation, a specific operation is executed or realized within the scope of the various aspects of the present disclosure.When introducing the elements of the various aspects of the present disclosure or its examples, the articles "a", "an", "the" and "said" are intended to represent the presence of one or more elements.The terms "include", "comprise" and "have" are intended to be inclusive, and mean that additional elements may also be present in addition to the listed elements.The term "exemplary" is intended to represent "example".The phrase "one or more of the following: A, B and C" means "at least one of A and / or at least one of B and / or at least one of C".
[0077] Having described various aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of various aspects of the disclosure as defined in the appended claims. As various changes may be made in the above-described constructions, products and methods without departing from the scope of various aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not restrictive.
Claims
1. A system, include: processor; as well as A computer storage medium storing instructions that, when executed by the processor, are operable to: separating, by the processor, each of a plurality of input segments of software code into a plurality of constituent blocks; generating an equivalent block using each of the first plurality of code generators for at least one of the plurality of building blocks, thereby generating a plurality of equivalent blocks corresponding to each of the code generators; for each code generator, determining a first decoding score for each equivalent block of the plurality of equivalent blocks corresponding to the code generator; aggregating, for each code generator, the first decoding scores across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting a second plurality of code generators from the first plurality of code generators based at least on a ranking of the first aggregated scores for the first plurality of code generators, the second plurality of code generators having a smaller count than the first plurality of code generators; as well as An output segment of software code is generated using each code generator of the second plurality of code generators.
2. The system of claim 1, wherein the instructions are further operable to: For each output segment of the software code, determining acceptance information; and For each code generator of the second plurality of code generators, an acceptance score is determined based at least on the acceptance information for the output segment of its software code.
3. The system of claim 1 , wherein determining the first decoding score include: replacing the constituent blocks with the equivalent blocks in the input segment of the software code from which the equivalent blocks were generated, thereby producing a modified segment; as well as A test result of testing the modified segment is determined.
4. The system of claim 3, wherein determining the test result include: A plurality of tests are performed on the modified segment, and wherein the test results comprise an aggregation of test results of the plurality of tests.
5. The system of claim 3, wherein determining the first decoding score further comprises: include: determining semantic similarity between the constituent blocks and the equivalent blocks; as well as The test result and the semantic similarity are aggregated into the first decoding score.
6. A method, include: separating, by the processor, each of the plurality of input segments of the software code into a plurality of constituent blocks; generating an equivalent block using each of the first plurality of code generators for at least one of the plurality of building blocks, thereby generating a plurality of equivalent blocks corresponding to each of the code generators; for each code generator, determining a first decoding score for each equivalent block of the plurality of equivalent blocks corresponding to the code generator; aggregating, for each code generator, the first decoding scores across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting a second plurality of code generators from the first plurality of code generators based at least on a ranking of the first aggregated scores for the first plurality of code generators, the second plurality of code generators having a smaller count than the first plurality of code generators; as well as An output segment of software code is generated using each code generator of the second plurality of code generators.
7. The method according to claim 6, further comprising: include: For each output segment of the software code, determining acceptance information; as well as For each code generator of the second plurality of code generators, an acceptance score is determined based at least on the acceptance information for the output segment of its software code.
8. The method of claim 6, wherein determining the first decoding score include: replacing the constituent blocks with the equivalent blocks in the input segment of the software code from which the equivalent blocks were generated, thereby producing a modified segment; as well as A test result of testing the modified segment is determined.
9. The method of claim 8, wherein determining the test result include: A plurality of tests are performed on the modified segment, and wherein the test results comprise an aggregation of test results of the plurality of tests.
10. The method of claim 8, wherein determining the first decoding score further comprises: include: determining semantic similarity between the constituent blocks and the equivalent blocks; as well as The test result and the semantic similarity are aggregated into the first decoding score.
11. One or more computer storage devices having computer executable instructions stored thereon, which, when executed by a computer, cause the computer to perform operations, the operations include: separating, by the processor, each of the plurality of input segments of the software code into a plurality of constituent blocks; generating an equivalent block using each of the first plurality of code generators for at least one of the plurality of building blocks, thereby generating a plurality of equivalent blocks corresponding to each of the code generators; for each code generator, determining a first decoding score for each equivalent block of the plurality of equivalent blocks corresponding to the code generator; aggregating, for each code generator, the first decoding scores across the plurality of equivalent blocks corresponding to the code generator into a first aggregate score for the code generator; selecting a second plurality of code generators from the first plurality of code generators based at least on a ranking of the first aggregated scores for the first plurality of code generators, the second plurality of code generators having a smaller count than the first plurality of code generators; as well as An output segment of software code is generated using each code generator of the second plurality of code generators.
12. The one or more computer storage devices of claim 11, wherein the operation further include: For each output segment of the software code, determining acceptance information; as well as For each code generator of the second plurality of code generators, an acceptance score is determined based at least on the acceptance information for the output segment of its software code.
13. The one or more computer storage devices of claim 11, wherein determining the first decoding score include: replacing the constituent blocks with the equivalent blocks in the input segment of the software code from which the equivalent blocks were generated, thereby producing a modified segment; as well as A test result of testing the modified segment is determined.
14. The one or more computer storage devices of claim 13, wherein determining the test result include: A plurality of tests are performed on the modified segment, and wherein the test results comprise an aggregation of test results of the plurality of tests.
15. The one or more computer storage devices of claim 13, wherein determining the first decoding score further comprises: include: determining semantic similarity between the constituent blocks and the equivalent blocks; as well as The test result and the semantic similarity are aggregated into the first decoding score.
