A method and apparatus for evaluating the quality of OCL sentence generation

Abstract

The embodiment of the present application provides a kind of method and device for evaluating OCL sentence generation quality, wherein, the scheme is through using BLEU improved Algorithm overall matching condition evaluation is carried out to OCL sentence, and K-Score is used as scoring standard, and the feature keyword matching condition evaluation is carried out to the OCL sentence, obtains overall evaluation score and keyword evaluation score.And the overall evaluation score and the keyword evaluation score are standardized according to the information amount contribution degree, so as to obtain the Key-Metric score of the OCL sentence.A better evaluation method is proposed, so that the generation result of OCL sentence can be better evaluated from multiple dimensions, so that the score can be more close to the actual quality of OCL sentence generation result.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for evaluating the quality of OCL statement generation. Background Technology

[0002] Object Constraint Language (OCL) is a formal specification language that holds significant importance in model-driven development and software modeling. Currently, in OCL research and code generation, using code generation technology to convert natural language into OCL statements has become a noteworthy issue. In the code generation process, evaluating the quality of the generated text is a crucial problem. A good evaluation standard can accurately reflect the transformation capability of code generation technology, pointing the way for further improvement; it can also simplify the research scenario, transforming complex generation problems into simple objective function optimization problems. Currently, the main evaluation method commonly used for code generation results is BLEU (Bilingual Evaluation Understudy).

[0003] However, in existing technologies, BLEU, as an indicator for evaluating machine translation, has unavoidable drawbacks when evaluating code generation results: it fails for excessively short results, assigning a score of 0 to a large number of excessively short generated results, even if the text is perfectly matched, it cannot correctly assess the quality of the generated code; and it ignores the special characteristics of the code, etc. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for evaluating the quality of OCL statement generation, to solve the technical problem in the prior art that the evaluation of code generation results fails for excessively short results and ignores the special characteristics of the code, leading to inaccurate evaluation results. The method includes:

[0005] Using BLEU improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score;

[0006] The K-Score was used as the scoring standard to evaluate the feature keyword matching of the OCL statement and obtain the keyword evaluation score.

[0007] Obtain the first standardized rule;

[0008] According to the first standardization rule, the overall evaluation score and the keyword evaluation score are standardized to obtain the Key-Metric score of the OCL statement.

[0009] In one embodiment, BLEU is used. improved The algorithm evaluates the overall matching performance of OCL statements, specifically including:

[0010] For the BLEU improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0. Here, N-gram is a set of N words, 1-gram is a set of 1 word, and K-gram is a set of K words.

[0011] Furthermore, the K-Score is used as the scoring standard to evaluate the feature keyword matching of the OCL statement and obtain a keyword evaluation score, specifically including:

[0012] Weighting is performed on different types of feature keywords to obtain the weight information of the feature keywords;

[0013] Extract the code feature keywords of the OCL statement, and determine the weight of the code feature keywords based on the weight information;

[0014] The weights are substituted into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement and obtain the keyword evaluation score.

[0015] Further, according to the first standardization rule, the overall evaluation score and the keyword evaluation score are standardized to obtain the Key-Metric score of the OCL statement, specifically including:

[0016] Based on the type, content, and weight of the code feature keywords, the information content value of the code feature keywords is obtained;

[0017] The Key-Metric score of the OCL statement is obtained by combining the information content value, the overall evaluation score, and the keyword evaluation score.

[0018] Furthermore, the Key-Metric score of the OCL statement is obtained by combining the information content value, the overall evaluation score, and the keyword evaluation score. Specifically, if the OCL statement cannot extract the code feature keywords, the overall evaluation score is output as the Key-Metric score of the OCL statement.

[0019] Furthermore, the Key-Metric score of the OCL statement is obtained by combining the information content value, the overall evaluation score, and the keyword evaluation score. The specific calculation method is as follows:

[0020] The formula for calculating the Key-Metric score is as follows:

[0021]

[0022] S B The overall evaluation score;

[0023] S k Assess scores for the keywords;

[0024] k i The ratio of the number of occurrences of the i-th keyword in the generated text to that in the reference text;

[0025] w i The weight of the code feature keyword;

[0026] v i Whether information is provided for the code feature keywords of the corresponding category.

[0027] This invention also provides an apparatus for evaluating the quality of OCL statement generation, addressing the technical problem in the prior art where evaluation of code generation results fails for excessively short results, and the inaccurate evaluation results are caused by ignoring the specific characteristics of the code. The apparatus includes:

[0028] The overall evaluation module is used for BLEU. improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score;

[0029] The local evaluation module is used to evaluate the feature keyword matching of the OCL statement using K-Score as the scoring standard, and obtain the keyword evaluation score.

[0030] The standardization module is used to obtain the first standardization rule;

[0031] The score output module is used to standardize the overall evaluation score and the keyword evaluation score according to the first standardization rule to obtain the Key-Metric score of the OCL statement.

[0032] In one embodiment, the overall evaluation module specifically includes:

[0033] The overall evaluation module uses BLEU. improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0.

[0034] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for evaluating the quality of OCL statement generation.

[0035] This invention also provides a computer-readable storage medium storing a computer program that performs any of the above-described methods for evaluating the quality of OCL statement generation.

[0036] Compared with the prior art, the beneficial effects that can be achieved by using at least one of the above-mentioned technical solutions in the embodiments of this specification include at least: using BLEU improved The algorithm evaluates the overall matching performance of OCL statements and uses K-Score as the scoring standard to evaluate the feature keyword matching performance of the OCL statements, obtaining an overall evaluation score and a keyword evaluation score. The overall evaluation score and the keyword evaluation score are then standardized to obtain the Key-Metric score of the OCL statement. BLEU improved Used to measure the overall matching of code, it can provide a good literal assessment of the quality of the generated code, avoiding misjudgments due to missing long grams, and effectively improving upon the shortcomings of BLEU in evaluating code generation results. Through BLEU improved Compared with K-Score scoring, this allows for a better evaluation of the generated results of OCL statements from multiple dimensions, making the scores more closely reflect the actual quality of the generated results. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic flowchart of a method for evaluating the quality of OCL statement generation provided by an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of a method for evaluating the quality of OCL statement generation provided in an embodiment of the present invention;

[0040] Figure 3 This is a computer device provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of a device for evaluating the quality of OCL statement generation, provided as an embodiment of the present invention.

[0042] The reference numerals in the figure are: memory 302, processor 304, device 400, overall evaluation module 401, local evaluation module 402, standardization module 403, and score output module 404. Detailed Implementation

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In this embodiment of the invention, a method for evaluating the quality of OCL statement generation is provided, such as... Figure 1 As shown, the method includes: Step S101: Using BLEU improved The algorithm evaluates the overall matching performance of the OCL statement and outputs an overall evaluation score; Step S102: Using K-Score as the scoring standard, the algorithm evaluates the feature keyword matching performance of the OCL statement and obtains a keyword evaluation score; Step S103: The algorithm obtains a first standardization rule; Step S104: The algorithm standardizes the overall evaluation score and the keyword evaluation score according to the first standardization rule to obtain the Key-Metric score of the OCL statement.

[0046] This embodiment uses BLEU improved The algorithm evaluates the overall matching performance of OCL statements and uses K-Score as the scoring standard to evaluate the feature keyword matching performance of the OCL statements, obtaining an overall evaluation score and a keyword evaluation score. The overall evaluation score and the keyword evaluation score are then standardized to obtain the Key-Metric score of the OCL statement. BLEU improved Used to measure the overall matching of code, it can provide a good literal assessment of the quality of the generated code, avoiding misjudgments due to missing long grams, and effectively improving upon the shortcomings of BLEU in evaluating code generation results. Through BLEU improvedCompared with K-Score scoring, this allows for a better evaluation of the generated results of OCL statements from multiple dimensions, making the scores more closely reflect the actual quality of the generated results.

[0047] Depend on Figure 1 As shown in the flowchart, the first embodiment of the present invention specifically includes the following steps:

[0048] Step S101: Using BLEU improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score;

[0049] In practice, BLEU is used. improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score for the BLEU. improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0. Here, N-gram is a set of N words, 1-gram is a set of 1 word, and K-gram is a set of K words.

[0050] Specifically, BLEU (Bilingual Evaluation Understudy) is a major evaluation method currently used for code generation results. For each input pair {Candidate (generated text), Reference (reference text)}, BLEU outputs an evaluation score BLEU∈[0,100], which measures the quality of the result. A higher score indicates that the generated result (Candidate) is closer to the reference answer (Reference), meaning higher quality, while a perfect score indicates complete similarity. The BLEU calculation formula is as follows:

[0051] BLEU = BP * S (Formula 1.1)

[0052] Where BP is the length penalty factor and S is the score. Since BLEU evaluates the proportion of matched N-grams in the generated text Candidate, a generated text Candidate that only correctly generates the Reference text can still receive a fairly high score; however, such a high score does not accurately reflect its generation quality. Therefore, the length penalty factor BP is introduced to avoid this problem. The value of BP is as follows:

[0053]

[0054] In Equation 1.2, l c Indicates the length of the generated text, l sThis indicates the effective length of the reference text. When the generated text length is shorter than the reference text, the penalty factor will adjust the N-gram score, lowering the score to more closely reflect the actual generated quality.

[0055] BLEU uses N-gram matching rules, which compare the similarity ratio of N groups of grams between the generated text and the reference text, and combine the scores under multiple N-grams to obtain the total score S. To avoid short sentences receiving excessively high scores, a length penalty factor BP is added to calculate the final score BLEU. The N-gram grammar model is one of the earliest statistical language models. In BLEU, sentences in the text are regarded as word sequences. A 1-gram is a word unit of length 1, usually a word; while an N-gram is a word unit of length N, composed of N 1-grams. In Equation 1.1, the score S is calculated as follows:

[0056]

[0057] In Equation 1.3, P n This represents the score for a gram of length n, measuring the occurrence of n-grams in the reference translation. SacreBLEU is an open-source implementation of the BLEU algorithm commonly used in Python. SacreBLEU calculates P-grams of lengths 1 to 4. n For each P n The matching score S for each sentence is obtained by sampling and summarizing using the geometric mean. n The calculation method is as follows:

[0058]

[0059] In Equation 1.4, let W k Let H represent the k-th n-gram. k (c i) W k In candidate translation C i The number of times H appears in H k (s ij ) represents W k In the standard answer S ij The number of times it appears in the list. Considering the recall count, we use min(H) k (c i ), H k (s ij The ')' represents the number of actual active grams in the candidate text. The gram W that appears m times in the Reference text... k The number of valid matches that can be found in the generated text Candidate should not exceed m.

[0060] In this embodiment, as Figure 2 As shown, based on the shortcomings of BLEU scoring, a Key-Metric evaluation is proposed, using an improved BLEU algorithm, namely the BLEU... improved The algorithm evaluates the overall matching performance and uses K-Score to evaluate keyword matching performance. The two are then weighted and combined to obtain a more comprehensive evaluation, Key-Metric. Key-Metric is a multi-dimensional evaluation method for code generation results proposed in this application, while K-Score is a feature keyword evaluation method for OCL statement generation results.

[0061] Furthermore, the BLEU improved This is an improvement on the BLEU algorithm. BLEU fails to evaluate short codes because it cannot extract longer grams from them, resulting in zero long grams that cannot be matched and thus receive incorrect scores. BLEU aims to describe the text matching degree of sentences at different levels when calculating N-grams, specifically evaluating long sentences at the character, word, and phrase levels. Considering that BLEU is typically used to evaluate long sentence matches, and that in this improvement, strings of length 1-gram and 2-gram still achieve character and phrase levels for short code pairs, this improvement, in principle, will not worsen BLEU's ability to evaluate short codes. improved When the results are short, only the matching of short grams can be used as a scoring factor, and the calculation method is shown in the following formula:

[0062]

[0063] In Equation 1.5, S B BLEU for the OCL statement improved Evaluation results; P n This represents the score for a gram of length N, measuring the occurrence of N-grams in the reference text. In practice, K is typically chosen with an upper bound of 4; for a normal BLEU score, at most gram matches of length 4 are evaluated. For improved BLEU... improved When the number of N-grams is 0, BLEU improved Only 1-gram to K-gram matching scores are evaluated. In actual experiments, a K value of 3 can handle most cases, discarding the 4-gram score; while a K value of 2 should handle all cases, using only 1-gram and 2-gram scores to evaluate sentences. BLEU improvedIt is used to measure the overall matching of the code, and can literally evaluate the quality of the generated code well, avoiding misjudgments due to missing long grams, and effectively improving the shortcomings of BLEU in evaluating code generation results.

[0064] Step S102: Use K-Score as the scoring standard to evaluate the feature keyword matching of the OCL statement and obtain the keyword evaluation score;

[0065] In practice, K-Score is used as the scoring standard to evaluate the feature keyword matching of the OCL statement and obtain a keyword evaluation score, which includes:

[0066] Step S201: Weight the feature keywords of different types to obtain the weight information of the feature keywords;

[0067] Step S202: Extract the code feature keywords of the OCL statement, and determine the weight of the code feature keywords based on the weight information;

[0068] Step S203: Substitute the weights into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement and obtain the keyword evaluation score.

[0069] Specifically, considering the uniqueness of code, this application proposes K-Score as a novel scoring sub-criteria to measure the matching of feature keywords within the code. In code, some literals are more important than others, carrying more information. Examples include `implies` (reserved words), `ooclIsTypeOf` (internal attributes), and `notEmpty` (library functions). We refer to these predefined literals as Keys, i.e., the feature keywords. Therefore, if the generated text `Candidate` contains all the corresponding feature keywords from the reference text `Reference`, its correctness has some basis; conversely, if the generated text `Candidate` contains none of the feature keywords from the reference text `Reference`, it is unlikely to be correct. Measuring the matching of these feature keywords in the result pairs allows for evaluation of the generated results from the perspective of code features, providing more information. Different feature keywords have different levels of importance depending on their type. For example, key words representing important relationships such as logical relationships, control flow relationships, invariants, and preconditions are reserved words, and it is almost impossible to express the same or even similar semantics through other key words; while key words representing related operations such as class attributes and inherited attributes are built-in attributes, and their semantics are not as strong as the former; library functions are in between the two, and there is room for them to be replaced by other operations.

[0070] Based on this, the embodiments of this application weight different types of feature keywords, and the types, contents, and weights of the feature keywords are shown in the table below:

[0071] Table 1

[0072]

[0073] Furthermore, the code feature keywords in the OCL statement are extracted, that is, the feature keywords in the OCL statement that overlap with the reference text. The weights of the code feature keywords are determined based on the weight information of the corresponding categories of feature keywords in Table 1. After obtaining the weights of the code feature keywords, these weights are substituted into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement, thereby obtaining the K-Score evaluation score of the OCL statement, that is, the keyword evaluation score. The K-Score calculation formula is shown below:

[0074]

[0075] Among them, S k The keyword in the OCL statement is evaluated with a score, i.e., a K-Score result; k i The ratio of the number of times the i-th feature keyword appears in the generated text Candidate to the number of times it appears in the reference text; w i The weights of the corresponding feature keywords; v i This indicates whether the feature keywords for the corresponding category provide information. If the frequency of the feature keywords for the corresponding category in the Reference is not 0, then the corresponding v i A value of 1 indicates that information was provided in this regard. In Equation 1.6 above, k i The calculation method is as follows:

[0076]

[0077] Where, count C (i) represents the number of times the i-th feature keyword appears in the generated text Candidate, count R (i) represents the number of times the i-th feature keyword appears in the reference text. The min function is min(count) C (i), count R The addition of (i) is to prevent the number of keyword occurrences in the generated text from exceeding the corresponding number in the reference text. The above calculation method uses the ratio of feature keywords in the generated text to those in the reference text, along with their corresponding weights and information content, to effectively measure keyword quality.

[0078] Step S103: Obtain the first standardized rule;

[0079] Step S104: Standardize the overall evaluation score and the keyword evaluation score according to the first standardization rule to obtain the Key-Metric score of the OCL statement.

[0080] Specifically, after calculating the overall evaluation score and keyword evaluation score for the OCL statement, and combining this with the above BLEU... improved The OCL code is comprehensively evaluated using the calculation results and K-Score results to obtain the Key-Metric score of the OCL statement. During the comprehensive Key-Metric scoring process, the calculation results are weighted according to the first standardization rule. Considering that not all reference text samples contain all three types of keywords, and more commonly, most samples contain only two or even one type of keyword, when a pair of evaluation samples contains only a few feature keywords in the reference text, the K-Score score provides relatively little information for the comprehensive Key-Metric score. Therefore, the weight of the K-Score score should be reduced when calculating the comprehensive Key-Metric score for the OCL statement generation results. Furthermore, according to the weighted classification of feature keywords, the information provided by more important feature keywords will have a greater weight in the comprehensive score. Therefore, by combining K-Score and BLEU... improved The information content is standardized according to the first standardization rule, so that the final Key-Metric score of the OCL statement is more accurate and objective.

[0081] In specific implementation, the overall evaluation score and the keyword evaluation score are standardized according to the first standardization rule to obtain the Key-Metric score of the OCL statement, specifically including:

[0082] Step S1041: Obtain the information content value of the code feature keyword based on the type, content, and weight of the code feature keyword;

[0083] Step S1042: Combine the information content value, the overall evaluation score, and the keyword evaluation score to obtain the Key-Metric score of the OCL statement.

[0084] Specifically, considering that the amount of information provided by the feature keywords varies in different evaluation results, the feature keywords are classified according to their weights. More important feature keywords will provide a larger proportion of information. The information content value of the code feature keywords is expressed as ∑w i v i Indicated. Wherein, w i V represents the weight of the corresponding feature keyword. i This indicates whether the feature keywords for the corresponding category provide information. If the frequency of the feature keywords for the corresponding category in the Reference code is not 0, then the corresponding v... i The value of 1 indicates that information was provided in this regard.

[0085] In specific implementation, the Key-Metric score of the OCL statement is obtained by combining the information content value, the overall evaluation score, and the keyword evaluation score, including:

[0086] The formula for calculating the Key-Metric score is as follows:

[0087]

[0088] Among them, S B The overall evaluation score;

[0089] S k Assess scores for the keywords;

[0090] k i The ratio of the number of occurrences of the i-th keyword in the generated text to that in the reference text;

[0091] w i The weight of the code feature keyword;

[0092] v i Whether information is provided for the code feature keywords of the corresponding category.

[0093] In specific implementation, the Key-Metric score of the OCL statement is obtained by combining the information content value, the overall evaluation score, and the keyword evaluation score. Specifically, if the OCL statement cannot extract the code feature keywords, the overall evaluation score is output as the Key-Metric score of the OCL statement.

[0094] Specifically, considering the varying content of feature keywords in the generated text of the OCL statements to be evaluated, when comprehensively scoring them, if the code feature keywords cannot be extracted from the OCL statement, then only the BLEU score of the OCL statement is used. improvedThe scoring result serves as the overall score, without applying a local K-Score. That is, ∑k in Equation 1.8 above... i v i The case where the value is 0. By considering the case where OCL statements lack characteristic keywords (Keys), the error in the overall scoring is reduced.

[0095] In this embodiment, a computer device is provided, such as... Figure 3 As shown, it includes a memory 302, a processor 304, and a computer program stored in the memory 302 and executable on the processor 304. When the processor 304 executes the computer program, it implements any of the above-described methods for evaluating the quality of OCL statement generation.

[0096] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0097] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described methods for evaluating the quality of OCL statement generation.

[0098] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0099] Based on the same inventive concept, this invention also provides an apparatus for evaluating the quality of OCL statement generation, as described in the following embodiments. Since the principle of an apparatus for evaluating the quality of OCL statement generation is similar to that of a method for evaluating the quality of OCL statement generation, implementation of such an apparatus can refer to implementation of a method for evaluating the quality of OCL statement generation, and will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0100] Figure 4 This is a structural block diagram of a device 400 for evaluating the quality of OCL statement generation according to an embodiment of the present invention, such as... Figure 4 As shown, it includes: an overall evaluation module 401 for using BLEU. improved The algorithm evaluates the overall matching performance of the OCL statement and outputs an overall evaluation score. A local evaluation module 402 uses K-Score as the scoring standard to evaluate the feature keyword matching performance of the OCL statement and obtains a keyword evaluation score. A standardization module 403 obtains a first standardization rule. A score output module 404 standardizes the overall evaluation score and the keyword evaluation score according to the first standardization rule to obtain the Key-Metric score of the OCL statement. This structure is described below.

[0101] In this embodiment, the overall evaluation module 401 uses the BLEU. improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0.

[0102] When the local evaluation module 402 obtains the keyword evaluation score, it weights different types of feature keywords to obtain the weight information of the feature keywords, extracts the code feature keywords of the OCL statement, determines the weight of the code feature keywords based on the weight information, substitutes the weight into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement, and obtains the keyword evaluation score.

[0103] The standardization module 403 is used to standardize the overall evaluation score and the keyword evaluation score according to the first standardization rule. This includes: obtaining the information content value of the code feature keywords based on their type, content, and weight; and combining the information content value, the overall evaluation score, and the keyword evaluation score to obtain the Key-Metric score of the OCL statement. If the OCL statement cannot extract the code feature keywords, the overall evaluation score is output as the Key-Metric score of the OCL statement.

[0104] The score output module 404 is used to output the Key-Metric score of the OCL statement according to the following Key-Metric score calculation formula.

[0105] The formula for calculating the Key-Metric score is as follows:

[0106]

[0107] S B The overall evaluation score;

[0108] S k Assess scores for the keywords;

[0109] k i The ratio of the number of occurrences of the i-th keyword in the generated text to that in the reference text;

[0110] w i The weight of the code feature keyword;

[0111] v i Whether information is provided for the code feature keywords of the corresponding category.

[0112] In another embodiment, software is also provided for executing the technical solutions described in the above embodiments and preferred embodiments.

[0113] In another embodiment, a storage medium is also provided, which stores the above-mentioned software. The storage medium includes, but is not limited to, optical discs, floppy disks, hard disks, and rewritable memory.

[0114] The embodiments of the present invention achieve the following technical effects: by using BLEU improvedThe algorithm evaluates the overall matching performance of OCL statements and uses K-Score as the scoring standard to evaluate the feature keyword matching performance of the OCL statements, obtaining an overall evaluation score and a keyword evaluation score. The overall evaluation score and the keyword evaluation score are then standardized to obtain the Key-Metric score of the OCL statement. BLEU improved Used to measure the overall matching of code, it can provide a good literal assessment of the quality of the generated code, avoiding misjudgments due to missing long grams, and effectively improving upon the shortcomings of BLEU in evaluating code generation results. Through BLEU improved Compared with K-Score scoring, this allows for a better evaluation of the generated results of OCL statements from multiple dimensions, making the scores more closely reflect the actual quality of the generated results.

[0115] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0116] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for evaluating the quality of OCL statement generation, characterized in that, include: Using BLEU improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score, whereby the BLEU score is... improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0. Here, N-gram is a set of N words, 1-gram is a set of 1 word, and K-gram is a set of K words. The K-Score is used as the scoring standard to evaluate the feature keyword matching of the OCL statement and obtain a keyword evaluation score. Different types of feature keywords are weighted to obtain the weight information of the feature keywords. The code feature keywords of the OCL statement are extracted, and the weight of the code feature keywords is determined based on the weight information. Substitute the weights into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement and obtain the keyword evaluation score. Obtain the first standardized rule; According to the first standardization rule, the overall evaluation score and the keyword evaluation score are standardized to obtain the Key-Metric score of the OCL statement, wherein... Based on the type, content, and weight of the code feature keywords, the information content value of the code feature keywords is obtained. Combining the information content value, the overall evaluation score, and the keyword evaluation score, the Key-Metric score of the OCL statement is obtained. If the OCL statement cannot extract the code feature keywords, then the overall evaluation score is output as the Key-Metric score of the OCL statement, including: The formula for calculating the Key-Metric score is as follows: S B The overall evaluation score; S k Assess scores for the keywords; k i The ratio of the number of occurrences of the i-th keyword in the generated text to that in the reference text; w i The weight of the code feature keyword; v i Whether information is provided for the code feature keywords of the corresponding category.

2. An apparatus for evaluating the quality of OCL statement generation, characterized in that, include: The overall evaluation module is used for BLEU. improved The algorithm evaluates the overall matching performance of OCL statements and outputs an overall evaluation score, whereby the BLEU score is... improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0. Here, N-gram is a set of N words, 1-gram is a set of 1 word, and K-gram is a set of K words. The local evaluation module is used to evaluate the feature keyword matching of the OCL statement using K-Score as the scoring standard to obtain a keyword evaluation score. Specifically, different types of feature keywords are weighted to obtain the weight information of the feature keywords. The module then extracts the code feature keywords of the OCL statement, determines the weight of the code feature keywords based on the weight information, and substitutes the weights into the K-Score calculation formula to evaluate the feature keyword matching of the OCL statement to obtain the keyword evaluation score. The standardization module is used to obtain the first standardization rule; The score output module is used to standardize the overall evaluation score and the keyword evaluation score according to the first standardization rule to obtain the Key-Metric score of the OCL statement, wherein... Based on the type, content, and weight of the code feature keywords, the information content value of the code feature keywords is obtained. Combining the information content value, the overall evaluation score, and the keyword evaluation score, the Key-Metric score of the OCL statement is obtained. If the OCL statement cannot extract the code feature keywords, then the overall evaluation score is output as the Key-Metric score of the OCL statement, including: The formula for calculating the Key-Metric score is as follows: S B The overall evaluation score; S k Assess scores for the keywords; k i The ratio of the number of occurrences of the i-th keyword in the generated text to that in the reference text; w i The weight of the code feature keyword; v i Whether information is provided for the code feature keywords of the corresponding category.

3. The apparatus for evaluating the quality of OCL statement generation according to claim 2, characterized in that: The overall evaluation module uses BLEU. improved The algorithm evaluates the matching scores from 1-gram to K-gram only when the number of N-grams in the OCL statement is 0.

4. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for evaluating the quality of OCL statement generation as described in claim 1.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs a method for evaluating the quality of OCL statement generation as described in claim 1.

Images