Model evaluation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a model evaluation method and device, equipment and a storage medium. The method comprises the following steps: acquiring a first code snippet, wherein the first code snippet comprises a plurality of function objects meeting a nesting constraint; determining a reference result of a function call associated with a target function object in the plurality of function objects; inputting test information to the first model, the test information instructing the first model to execute a function call associated with the target function object based on a test code, the test code comprising a first code snippet and at least one second code snippet; and generating evaluation information of the first model based on comparison of an execution result output by the first model and a reference result. Based on the mode, at least one second code snippet can be added into the first code snippet to form the test code, the complexity of the test code is increased, the test code is closer to a code in an actual code environment, and the first model can be evaluated more accurately.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to a model evaluation method, apparatus, device, and computer-readable storage medium. Background Art

[0002] With the advancement of technology, model capabilities are showing increasing potential in many fields, especially in program development scenarios such as integrated development environments (IDEs). How to evaluate model capabilities is a key issue. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for model evaluation is provided. The method includes: obtaining a first code snippet, the first code snippet including multiple function objects that satisfy a nesting constraint; determining a reference result of a function call associated with a target function object among the multiple function objects; inputting test information into a first model, the test information instructing the first model to execute the function call associated with the target function object based on test code, the test code including the first code snippet and at least one second code snippet; and generating evaluation information for the first model based on a comparison of an execution result output by the first model with the reference result.

[0004] In a second aspect of the present disclosure, a device for model evaluation is provided. The device includes: an acquisition module configured to acquire a first code snippet, the first code snippet including multiple function objects that satisfy nesting constraints; a determination module configured to determine a reference result of a function call associated with a target function object among the multiple function objects; an input module configured to input test information into a first model, the test information instructing the first model to execute the function call associated with the target function object based on a test code, the test code including the first code snippet and at least one second code snippet; and a generation module configured to generate evaluation information for the first model based on a comparison between an execution result output by the first model and the reference result.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 shows a flow chart of a model evaluation process according to some embodiments of the present disclosure;

[0011] Figure 3 shows an example flow chart of model evaluation according to some embodiments of the present disclosure;

[0012] Figure 4 shows a schematic structural block diagram of an apparatus for model evaluation according to certain embodiments of the present disclosure;

[0013] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0017] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0018] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0019] Embodiments of the present disclosure propose a model evaluation scheme. According to the scheme, a first code snippet is obtained, the first code snippet including multiple function objects that satisfy nesting constraints; a reference result of a function call associated with a target function object among the multiple function objects is determined; test information is input into a first model, the test information instructing the first model to execute the function call associated with the target function object based on test code, the test code including the first code snippet and at least one second code snippet; and evaluation information for the first model is generated based on a comparison of the execution result output by the first model with the reference result.

[0020] Based on this approach, the embodiments of the present disclosure can add at least one second code snippet to the first code snippet to form a test code, which increases the complexity of the test code and makes the test code closer to the code in the actual code environment, helping to evaluate the first model more accurately.

[0021] Sample Environment

[0022] Figure 11 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0023] In the example environment 100, the electronic device 110 is configured to determine scoring information for a first model 120. The scoring information may be associated with the first model's ability to execute a function call associated with a predetermined function object. The first model 120 may be any suitable machine learning model, which may be deployed on any suitable device, such as the electronic device 110 (not shown), or on other devices or servers.

[0024] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal with a display device, including a mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, PDA, portable game terminal, VR / AR device, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, game device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the target user (such as "wearable" circuit, etc.).

[0025] The electronic device 110 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0026] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0027] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0028] Example Process

[0029] Figure 2 FIG2 shows a flow chart of a model evaluation process 200 according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. Figure 1 Process 200 is described.

[0030] In block 210 , the electronic device 110 obtains a first code snippet, where the first code snippet includes a plurality of function objects that satisfy a nesting constraint.

[0031] In some embodiments, the first code snippet can be configured to implement a specific function or algorithm. In some embodiments, a function object represents an object having functional behavior, which can be called. The number of the multiple function objects included in the first code snippet can be set as needed, for example, it can be set to 3. For each function object included in the first code snippet, the function object can call other function objects, and of course the function object can also be called by other function objects. The specific calling relationship of these multiple function objects can also be set as needed.

[0032] In some embodiments, multiple function objects indicate the execution of at least one operation from a preset set of operations. This preset set of operations may include any appropriate operation, such as, but not limited to, arithmetic operations, trigonometric operations, loop structure operations, recursive function operations, conditional statement operations, exponential operations, and logarithmic operations. The at least one operation may be any operation from the aforementioned set of predefined operations, and the operations indicated by the multiple function objects may be the same or different.

[0033] In order to make the first model focus more on reasoning ability (ie, the first model determines the function call ability of executing a certain function object) rather than computing ability, as an example, these multiple function objects can all indicate a simple algorithm such as executing four arithmetic operations.

[0034] Take the first code snippet below as an example:

[0035]

[0036] It can be seen from the first code snippet of the above example that the first code snippet of the example includes three function objects, namely func_4253, func_2214 and func_6345, and func_6345 calls func_2214, and the three function objects func_4253, func_2214 and func_6345 all indicate the execution of four arithmetic operations. Specifically, func_4253 indicates the execution of addition operation, func_2214 indicates the execution of multiplication operation, and func_6345 indicates the execution of subtraction operation.

[0037] In some embodiments, nesting constraints may indicate that multiple function objects must comply with specific rules or restrictions when calling each other. For example, nesting constraints may indicate various constraints such as the number of nesting levels corresponding to the multiple function objects, the calling order, parameter passing, and return value processing. The nesting level represents the depth or hierarchy of function calls in the call chain formed by the multiple function objects.

[0038] Since the nesting level of code is related to the complexity of the code, the deeper the nesting level, the higher the complexity of the corresponding code. Therefore, in order to obtain code snippets of different complexities, the electronic device 110 can construct code snippets of different nesting levels. The nesting level refers to the depth of the call chain formed by the mutual calls between these multiple function objects.

[0039] Due to the reasoning process of the model, when it comes to the nesting of function objects, a large degree of differentiation can be reflected within 10 layers of nesting. 30-40 layers of nesting can simulate the performance of the model in a high-nesting scenario. Therefore, the nesting constraint can indicate that the number of nesting levels corresponding to multiple function objects is within a predetermined range. The predetermined range can be any appropriate range, for example, it can be set to between [2,40].

[0040] Take the first code snippet below as an example:

[0041]

[0042] It can be seen from the first code snippet of the above example that the first code snippet of the example includes three function objects, namely func_4253, func_2214 and func_6345, and func_6345 calls func_2214, which in turn calls func_4253. Therefore, the number of nested levels corresponding to these three function objects is 3.

[0043] In block 220 , the electronic device 110 determines a reference result of a function call associated with a target function object among the plurality of function objects.

[0044] In some embodiments, the target function object may be the function object corresponding to the maximum embedding level among the multiple function objects. The function object at the maximum embedding level may also be referred to as the top-level function object or the external function object, which is the outermost function called.

[0045] In some embodiments, the function calls associated with a target function object represent calls to all function objects that are directly or indirectly related to the target function object.

[0046] In some embodiments, the reference result represents the expected result of the function call associated with the target function object and is used to verify whether the function call is executed correctly.

[0047] In some embodiments, the electronic device 110 may determine a target function object corresponding to a maximum embedding level from a plurality of function objects, that is, determine a top-level function object from the plurality of function objects as the target function object.

[0048] Furthermore, the electronic device 110 may construct a function call associated with the target function object based on the reference variable value. The reference variable value may be any appropriate variable, such as any appropriate random number. For example, if the target function object is func_6345, the function call may be func_6345(2).

[0049] Furthermore, the electronic device 110 can utilize the second model to process the function call based on the first code snippet to generate a reference result. The second model can be any appropriate machine learning model, which will not be described in detail here. For example, if func_6345(2)=9, then this 9 is the reference result.

[0050] For ease of description, func_6345(2) can also be called the question, and 9 can be called the answer generated by calling the objective function object func_6345.

[0051] In block 230 , the electronic device 110 inputs test information to the first model 120 , where the test information instructs the first model 120 to execute a function call associated with a target function object based on a test code, where the test code includes a first code snippet and at least one second code snippet.

[0052] In some embodiments, the first model 120 may be any appropriate machine learning model. The first model 120 may be deployed on the electronic device 110 or on other devices.

[0053] In some embodiments, in order to simulate a complex code environment in the real world, the electronic device 110 can construct a test code based on a first code snippet and at least one second code snippet. The at least one second code snippet can also be referred to as the context code corresponding to the first code snippet, which is an interference code snippet used to increase the complexity of the first model 120 executing the function call associated with the target function object based on the test code. Specifically, since the at least one second code snippet is added to the test code, the corresponding test code is more complex than the test code that only includes the first code snippet itself, and the degree of test on the reasoning ability of the first model 120 is also greater.

[0054] As an example, the electronic device 110 may insert the at least one second code snippet into the first code snippet to obtain a test code corresponding to a preset context length. The preset context length may be any appropriate length, such as ensuring that the size of a token converted from the test code is 16k, 32k, 64k, 128k, or the like.

[0055] In some embodiments, the at least one second code snippet may include at least one randomly generated function object. The at least one function object may also indicate the execution of at least one operation in a preset operation set. The preset operation set may include any appropriate operation, such as but not limited to four arithmetic operations, trigonometric function operations, loop structure operations, recursive function operations, conditional statement operations, exponential operations, and logarithmic operations, etc. The at least one operation may be any operation in the above-mentioned budget operation set, and the operations indicated by the at least one function object may be the same or not completely the same. As an example, the at least one function object may indicate the execution of a simple algorithm such as four arithmetic operations.

[0056] It should be noted that there is no calling relationship between the at least one function object included in the second code snippet and the multiple function objects included in the first code snippet.

[0057] Since the first model 120 has different capabilities in processing contexts of different lengths, the present disclosure can obtain test codes of different lengths by setting second code snippets corresponding to different token lengths, thereby ensuring that the code reasoning capabilities of the first model 120 in contexts of different lengths can be comprehensively evaluated.

[0058] It should be noted that, since the number of tokens consumed by the first model 120 includes multiple stages such as model input, model calculation, and model output, the present disclosure can determine the token length consumed by the first model 120 during the calculation and output processes based on a predetermined method. Furthermore, the electronic device 110 can determine the target token length based on the expected token length and the consumed token length. Furthermore, the electronic device 110 can construct a test code based on a second code snippet of the target token length to leave a spare consumable token length for the calculation and output of the first model 120.

[0059] In some embodiments, the at least one second code snippet may be inserted into any appropriate position in the first code snippet, which will not be described in detail herein.

[0060] The following uses specific code examples to illustrate the test code construction process.

[0061] Take the first code snippet below as an example:

[0062]

[0063] It can be seen from the first code snippet of the above example that the first code snippet of this example includes three function objects, namely func_4253, func_2214 and func_6345.

[0064] If the at least one second code fragment includes two function objects, func_1233 and func_6235, the test code may be the following example code:

[0065]

[0066] It can be seen from the above test code examples that the present disclosure adds a second code snippet corresponding to func_1233 and a second code snippet corresponding to func_6235 to the first code snippet.

[0067] Furthermore, electronic device 110 may construct test information based on the test code, description information of the function call associated with the target function object, and identification information of the target function object. The test information is used to instruct first model 120 to execute the function call associated with the target function object based on the test code. Test information may also be referred to as a prompt or prompt information, which will not be described in detail here.

[0068] Specifically, the electronic device 110 can fill the identification information of the target function object, the description information of the function call, and the test code into the prompt information template to construct the test information. The prompt information template can indicate a predetermined filling prompt or filling format to indicate that the identification information of the target function object, the description information of the function call, and the test code are filled into the corresponding positions in the prompt information template to construct the test information. The identification information of the target function object can be any appropriate information that can characterize the identity of the target function object, such as the name of the target function object. The description information of the function call can be any appropriate information, such as input parameters, the function's function, etc.

[0069] In some embodiments, to facilitate viewing, the test information may further indicate the location of the execution result output by the first model 120 within the output of the first model 120. For example, the execution result may be indicated as being presented at the end of the output. In addition to the execution result, the output may also include intermediate results and log information generated during the execution process, which are not detailed here.

[0070] In block 240 , the electronic device 110 generates evaluation information of the first model 120 based on a comparison between the execution result output by the first model 120 and the reference result.

[0071] In some embodiments, if the comparison result between the execution result and the reference result indicates that the execution result matches the reference result, it is more conducive to the evaluation of the first model 120, that is, the first model has a stronger corresponding reasoning ability.

[0072] In some embodiments, the electronic device 110 can use a test data set to test the processing capability of the first model 120. The test data set can include a test sample. Specifically, the electronic device 110 can construct a test sample based on the test code, function call, and reference results.

[0073] In order to improve the accuracy of model evaluation, in some embodiments, the test sample may also include reference information. In some embodiments, the reference information may include but is not limited to at least one of the following: the code language corresponding to the test code, the identification information corresponding to the test sample, the function nesting level corresponding to the test code, the context length corresponding to the test code, and the number of function objects included in the test code. The code language can be any appropriate language, such as python, java, etc. The identification information corresponding to the test sample can be any appropriate information that characterizes the identity of the test sample, such as the ID corresponding to the test sample. The function nesting level corresponding to the test code can be the function nesting level of multiple function objects included in the test code. The context length corresponding to the test code can be the token length corresponding to the test code.

[0074] In some embodiments, these reference information are used to assist in analyzing the reasoning capabilities of the first model 120 in at least one of multiple dimensions, such as different nesting depths, corresponding to different numbers of function objects, indicating different numbers of function objects for predetermined operations, or corresponding to different context lengths.

[0075] As an example, the evaluation information can indicate the processing capability of the first model 120 for function calls associated with different nesting levels. The function call associated with a certain nesting level represents that the first model 120 executes a function call associated with a target function object based on a certain test code, and the multiple function objects included in the first target code snippet in this test code correspond to this nesting level, and this test code can also include a randomly inserted second target code snippet (used to interfere with the first model execution of the function call associated with the target function object), and this target function object is the top-level function object among the multiple function objects included in the first target code snippet. For example, the evaluation information can indicate the processing capability of the first model 120 for function calls associated with 3 nesting levels. The evaluation information can also indicate the processing capability of the first model 120 for function calls associated with 4 nesting levels, and so on.

[0076] As another example, the evaluation information can also indicate the processing capability of the first model 120 for function calls associated with different context lengths. The function call associated with a certain context length represents that the first model 120 executes a function call associated with a target function object based on a certain test code, and the second target code snippet included in the test code (used to interfere with the first model's execution of the function call associated with the target function object) corresponds to the context length, and the first target code snippet included in the test code includes the target function object, which is the top-level function object among the multiple function objects included in the first target code snippet. For example, the evaluation information can indicate the processing capability of the first model 120 for function calls associated with a context length of 16K. The evaluation information can also indicate the processing capability of the first model 120 for function calls associated with a context length of 32K, and so on.

[0077] In some embodiments, the electronic device 110 may associate the result of the comparison with at least one tag associated with the test information. The result indicates whether the execution result output by the first model 120 matches the reference result. The at least one tag may include but is not limited to the code language corresponding to the test code, the function nesting level corresponding to the test code, the context length corresponding to the test code, the number of function objects included in the test code, and the like. As an example, for a test code with a context length of 16K, if the comparison result indicates that the execution result determined by the first model 120 based on the test information constituted by the test code matches the corresponding reference result, then the comparison result is associated with the tag "context length is 16K".

[0078] Furthermore, the electronic device 110 may generate evaluation information for the first model 120 based on test statistics associated with at least one tag. For example, if, under the tag "context length is 16K," there are 6 comparison results indicating that the execution result matches the reference result, while there are 94 comparison results indicating that the execution result does not match the reference result, then the evaluation information for the first model 120 may indicate that the first model 120 has poor reasoning capabilities for test samples with a context length of 16K.

[0079] In some embodiments, the electronic device 110 can also use the same test data set to test the differences in function call capabilities of different models. Specifically, for each model, the electronic device 110 can determine the target number of test samples in the test data set in which the execution result output by the first model matches the reference result. Furthermore, the electronic device 110 can compare the target number corresponding to each model to determine the difference in the call capability of each model under this test data set.

[0080] Figure 3 An example flow chart of model evaluation according to some embodiments of the present disclosure is shown. Figure 3 Provide explanation.

[0081] In block 301 , the electronic device 110 generates a function call chain.

[0082] As an example, the electronic device 110 may construct multiple function objects, and these multiple function objects may generate a function call chain by calling each other.

[0083] As an example, the multiple function objects may indicate the execution of four arithmetic operations.

[0084] As an example, the multiple function objects may satisfy a nesting constraint. The nesting constraint may indicate that the number of nesting levels corresponding to the multiple function objects is within a predetermined range. The predetermined range may be any appropriate range, such as [2, 40]. That is, the call depths corresponding to the multiple function objects may be between [2, 40].

[0085] Furthermore, the electronic device 110 may construct a first code segment based on the multiple function objects.

[0086] At block 302 , the electronic device 110 generates an interference function object.

[0087] As an example, the electronic device 110 may construct a function object set, which may include a plurality of interference function objects. The interference function objects are function objects used to interfere with the first model 120 to execute corresponding function calls.

[0088] As an example, the electronic device 110 may store all interference function objects included in the function object set based on a list.

[0089] In block 303 , the electronic device 110 performs shuffle processing on the interference functions, and concatenates the shuffled interference functions with multiple function objects corresponding to the function call chain.

[0090] As an example, the electronic device 110 may randomly shuffle the order of the functions in the list. Further, the electronic device 110 may construct at least one second code snippet based on a group of function objects in the randomly shuffled list.

[0091] In some embodiments, the set of function objects is used to interfere with the first model 120 to execute a function call associated with a target function object, which is a top-level function object of the plurality of function objects included in the first code snippet.

[0092] As an example, the tokon corresponding to this set of function objects can correspond to any appropriate length.

[0093] Furthermore, the electronic device 110 may insert the at least one second code segment into the first code segment to obtain a test code.

[0094] In block 304 , the electronic device 110 generates questions and model answers.

[0095] As an example, the electronic device 110 may determine a target function object corresponding to the maximum embedding level among the multiple function objects in the first code snippet, i.e., a top-level function object. Further, the electronic device 110 may use random numbers to construct a function call associated with the target function object to generate a question.

[0096] As an example, the electronic device 110 may utilize the second model to process the function call based on the first code snippet to generate a reference result, that is, generate a standard answer. The second model may be any appropriate machine learning model.

[0097] At block 305 , the electronic device 110 constructs scoring logic.

[0098] As an example, the electronic device 110 can be set to add one point to the score of the first model 120 every time the execution result output by the first model 120 is consistent with the corresponding reference result, and set not to adjust the score of the first model 120 when the execution result is inconsistent with the corresponding reference result.

[0099] In block 306 , the electronic device 110 constructs a question bank.

[0100] As an example, the electronic device 110 may construct a test sample based on the test code, questions, and answers, and then add the test sample to the question bank.

[0101] Furthermore, the electronic device 110 can determine the scoring information of the first model 120 based on the question bank. Specifically, the electronic device 110 can determine, for each test sample in the question bank, whether the execution result output by the first model 120 based on the test sample is consistent with the corresponding answer in the test sample. Furthermore, the electronic device 110 can accumulate the number of test samples with consistent answers and the number of test samples with inconsistent answers to determine the scoring information of the first model 120.

[0102] Based on this approach, the embodiments of the present disclosure can add at least one second code snippet to the first code snippet to form a test code, which increases the complexity of the test code and makes the test code closer to the code in the actual code environment, helping to evaluate the first model more accurately.

[0103] Example devices and equipment

[0104] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 : A schematic structural block diagram of an apparatus 400 for model scoring according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in the electronic device 110 discussed above. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0105] like Figure 4 As shown, the device 400 includes an acquisition module 410, which is configured to acquire a first code snippet, the first code snippet includes multiple function objects that satisfy the nesting constraint; a determination module 420, which is configured to determine a reference result of a function call associated with a target function object among the multiple function objects; an input module 430, which is configured to input test information into the first model, the test information instructing the first model to execute the function call associated with the target function object based on the test code, the test code including the first code snippet and at least one second code snippet; and a generation module 440, which is configured to generate evaluation information of the first model based on a comparison between the execution result output by the first model and the reference result.

[0106] In some embodiments, the apparatus 400 further includes an inserting module configured to: insert at least one second code segment into the first code segment to obtain a test code corresponding to a preset context length.

[0107] In some embodiments, the at least one second code snippet includes at least one randomly generated function object.

[0108] In some embodiments, the determination module 420 is further configured to determine the target function object corresponding to the maximum embedding level from multiple function objects; construct a function call associated with the target function object based on the reference variable value; and use the second model to process the function call based on the first code snippet to generate a reference result.

[0109] In some embodiments, the apparatus 400 further includes a filling module configured to fill the identification information of the target function object, the description information of the function call, and the test code into the prompt information template to construct the test information.

[0110] In some embodiments, the device 400 also includes a construction module, which is configured to: construct a test sample in a test data set based on the test code, function calls and reference results, and the test sample also includes reference information, and the reference information includes at least one of the following: the code language corresponding to the test code; the identification information corresponding to the test sample; the function nesting level corresponding to the test code; the context length corresponding to the test code; and the number of function objects included in the test code.

[0111] In some embodiments, the apparatus 400 further includes a generating module 440 , further configured to generate evaluation information of the first model based on test statistics information associated with at least one label.

[0112] In some embodiments, the evaluation information indicates at least one of: a processing capability of the first model for function calls associated with different nesting levels; and a processing capability of the first model for function calls associated with different context lengths.

[0113] In some embodiments, the nesting constraint indicates that a number of levels of nesting levels corresponding to the plurality of function objects is within a predetermined range.

[0114] In some embodiments, the plurality of function objects indicates execution of at least one operation in a preset set of operations.

[0115] The units included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0116] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The illustrated electronic device 500 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 An electronic device 110 is shown.

[0117] like Figure 5 As shown, electronic device 500 is in the form of a general electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 500.

[0118] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 500.

[0119] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0120] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0121] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0122] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0123] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0125] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0126] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0127] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for model evaluation, comprising: Obtaining a first code snippet, wherein the first code snippet includes a plurality of function objects that satisfy a nesting constraint; determining a reference result of a function call associated with a target function object among the plurality of function objects; Inputting test information into the first model, the test information instructing the first model to execute the function call associated with the target function object based on test code, the test code including the first code snippet and at least one second code snippet; as well as Based on a comparison between the execution result output by the first model and the reference result, evaluation information of the first model is generated.

2. The method according to claim 1, further comprising: Inserting the at least one second code fragment into the first code fragment to obtain the test code corresponding to a preset context length. The method according to claim 1 , wherein the at least one second code snippet comprises at least one randomly generated function object.

4. The method of claim 1 , wherein determining a reference result of a function call associated with a target function object among the plurality of function objects comprises: determining the target function object corresponding to the maximum embedding level from the plurality of function objects; constructing the function call associated with the target function object based on the reference variable value; as well as The function call is processed based on the first code snippet using a second model to generate the reference result.

5. The method according to claim 1, further comprising: The identification information of the target function object, the description information of the function call and the test code are filled into a prompt information template to construct the test information.

6. The method according to claim 1, further comprising: A test sample in a test data set is constructed based on the test code, the function call, and the reference result, wherein the test sample further includes reference information, and the reference information includes at least one of the following: The code language corresponding to the test code; identification information corresponding to the test sample; The function nesting level corresponding to the test code; The context length corresponding to the test code; The number of function objects included in the test code.

7. The method according to claim 1, wherein generating evaluation information of the first model based on comparison of the execution result output by the first model with the reference result comprises: associating a result of the comparison to at least one tag associated with the test information, the result indicating whether the execution result output by the first model matches the reference result; as well as The evaluation information of the first model is generated based on test statistics associated with the at least one label.

8. The method according to claim 1, wherein the evaluation information indicates at least one of the following: the processing capability of the first model for function calls associated with different nesting levels; The first model is directed to processing capabilities of function calls associated with different context lengths. 9 . The method according to claim 1 , wherein the nesting constraint indicates that a number of levels of nesting levels corresponding to the plurality of function objects is within a predetermined range.

10. The method according to claim 1, wherein the plurality of function objects indicate execution of at least one operation in a preset operation set.

11. A device for model evaluation, comprising: An acquisition module is configured to acquire a first code snippet, wherein the first code snippet includes a plurality of function objects that satisfy a nesting constraint; a determining module configured to determine a reference result of a function call associated with a target function object among the plurality of function objects; an input module configured to input test information into the first model, wherein the test information instructs the first model to execute the function call associated with the target function object based on a test code, wherein the test code includes the first code snippet and at least one second code snippet; as well as The generating module is configured to generate evaluation information of the first model based on a comparison between the execution result output by the first model and the reference result.

12. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

Images