Evaluation method and device, electronic equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2024-11-29
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, task evaluation relying on artificial intelligence models is inefficient, costly, and lacks standardization and normalization.
By creating evaluation tasks and associating them with target tasks, evaluation results are generated using artificial intelligence models, reducing human intervention. Evaluation results are generated based on evaluation criteria, thereby reducing evaluation costs and ensuring the standardization and normalization of evaluation.
It improves evaluation efficiency, reduces evaluation costs, ensures the standardization and normalization of evaluation, and reduces the influence of subjective human judgment.
Smart Images
Figure CN122152647A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more particularly to an evaluation method, apparatus, electronic device, and storage medium. Background Technology
[0002] In the field of computer science, tasks are typically defined, which can be performing certain methods or implementing certain functions; for example, a task could be translating text. These tasks can be implemented using artificial intelligence models.
[0003] After a task is created, it is usually necessary to evaluate the functions performed by the task. This typically relies on personnel with professional knowledge to evaluate whether the task's performance meets expectations based on their experience and subjective judgment. Summary of the Invention
[0004] This disclosure provides an evaluation method, apparatus, electronic device, and storage medium.
[0005] The following technical solution is adopted in this disclosure.
[0006] In some embodiments, this disclosure provides an evaluation method, including:
[0007] Create an evaluation task;
[0008] Establish the association between the evaluation task and the target task to be evaluated;
[0009] In response to an event that the evaluation task is executed, the associated data of the target task is input into the evaluation task so that an evaluation result is generated by an artificial intelligence model based on the evaluation criteria.
[0010] The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
[0011] In some embodiments, this disclosure provides an evaluation apparatus, including:
[0012] Create a unit to create evaluation tasks;
[0013] The creation unit is also used to establish the association between the evaluation task and the target task to be evaluated;
[0014] A control unit is configured to respond to an event that triggers the evaluation task by inputting the associated data of the target task into the evaluation task, so as to generate evaluation results based on the evaluation criteria through an artificial intelligence model.
[0015] The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
[0016] In some embodiments, this disclosure provides an electronic device, including: at least one memory and at least one processor;
[0017] The memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above method.
[0018] In some embodiments, this disclosure provides a computer-readable storage medium for storing program code that, when run by a processor, causes the processor to perform the methods described above.
[0019] The evaluation method provided in this disclosure eliminates the need for manual evaluation of the target task, thereby improving evaluation efficiency, reducing evaluation costs, and ensuring that the evaluation criteria remain unchanged, thus guaranteeing the standardization and normalization of the evaluation. Attached Figure Description
[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.
[0021] Figure 1 This is a flowchart of the evaluation method according to an embodiment of the present disclosure.
[0022] Figure 2 This is a schematic diagram illustrating the creation of an evaluation task according to an embodiment of this disclosure.
[0023] Figure 3 This is a schematic diagram of the evaluation task in an embodiment of this disclosure.
[0024] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation
[0025] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0027] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0028] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0029] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0031] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0032] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0034] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.
[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0036] The solutions provided by the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.
[0037] like Figure 1 As shown, Figure 1 This is a flowchart of an evaluation method according to an embodiment of the present disclosure, which includes the following steps.
[0038] S11: Create an evaluation task.
[0039] S12: Establish the association between the evaluation task and the target task to be evaluated.
[0040] In some embodiments, the method proposed in this disclosure can be used in terminals such as mobile phones, tablets, and computers. The process of creating an evaluation task may include: determining the evaluation criteria. The target task to be evaluated is the task from which the data to be evaluated comes from. For example... Figure 2 As shown, it schematically illustrates the process of creating an evaluation task. This could be achieved by triggering a control in the client to create the evaluation task, which then displays a diagram like... Figure 2The interface shown allows you to set the key (identifier of the assessment task), version, etc., for the created assessment task. In this interface, you can select the first prompt word, the expected output data of the target task (reference answer), the assessment criteria (scoring standards), and the format of the assessment task's output (expected output). Then, trigger the "OK" button to complete the creation of the assessment task. After creating the assessment task, you can establish its association with the target task. Specifically, triggering the button for the target task will display a selection interface to choose which task to use for assessing the target task. In this selection interface, you can choose the previously created assessment task, thus establishing the association between the two. The target task can be, for example, translating sentences, generating an article summary, or continuing an article. The target task can be a function implemented through an artificial intelligence program. The input data for the target task can be the data required to execute the target task; for example, when generating an article summary, the input data can be the article itself. The output data of the target task can be the data generated after executing the target task; for example, if the target task is to generate an article summary, the output data is the summary generated in a particular execution of the target task. Evaluation criteria may include scoring standards and reference answers used for scoring. During the execution of the evaluation task, the input data (the output data of the target task is usually the input data) is evaluated to obtain evaluation results. The evaluation task is used by an artificial intelligence model to generate evaluation results for the input data based on the evaluation criteria. The artificial intelligence model can be a large model.
[0041] S12: In response to the event of executing the evaluation task, the associated data of the target task is input into the evaluation task so as to generate evaluation results by means of an artificial intelligence model based on the evaluation criteria.
[0042] In some embodiments, the associated data of the target task includes the input data and output data of the target task. The input and output data can be the input data and corresponding output data for each execution of the target task. After creating the evaluation task, some operations can be performed to execute the evaluation task. For example, a button to run the evaluation task can be triggered by clicking, which generates an event for executing the evaluation task. Alternatively, pre-set conditions for the evaluation task to be executed can be used, and the event for executing the evaluation task is triggered when the given conditions are met. When the event for executing the evaluation task occurs, the pre-set input and output data of the target task are input into the evaluation task. The evaluation task then uses an artificial intelligence program to evaluate the input and output data of the target task in this instance to obtain the evaluation result. In some embodiments, the target task generates output data based on input data. The input data can be pre-defined and selected from an input set. The input set can have a corresponding output set, where the data in the output set represents the expected output data corresponding to each input data point. The expected output data is the ideal output data obtained by the target task after processing the input data. Therefore, in some embodiments, the evaluation task can include expected output data corresponding to the input data. When the evaluation task is executed multiple times, the expected output data can be different each time, depending on the input data of the target task. That is, the expected output data can be determined based on the input data of the target task, and the expected output data can change as the input data changes.
[0043] To better illustrate the method proposed in this disclosure, a specific embodiment is given below, in conjunction with the accompanying drawings. Figure 3 This section provides an explanation. The target task can be a question or problem to be solved, or it can be a function implemented through an artificial intelligence program. For example, a function might be developed to generate an answer based on a user-input question. When this function is executed, it performs the target task. In other words, the target task can be implemented through code, more specifically, instantiated interface code. After the target task is created, its corresponding evaluation task can be created. Figure 3 The illustration shows an example of creating an evaluation task. The evaluation task uses a first prompt to describe its function. When creating the task, the evaluation criteria can be selected. During execution, the placeholders in the first prompt are replaced with actual values, resulting in the instantiated first statement of the first prompt. This first statement specifically tells the AI program how to generate the evaluation results. The evaluation task itself can also be a function, meaning it can be implemented through code. After creation, the task can be executed. During execution, the target task can obtain pre-set input data, generate output data, and input both the input and output data into the evaluation task, which then generates the evaluation results. Evaluation criteria can be set within the task (e.g., evaluation standards). Figure 3 The scoring criteria and reference outputs (e.g.) Figure 3 The reference answer (ref) in the code, the scoring criteria and reference output can be displayed as placeholders (such as parameter variables), which makes it easier to change and reduces code coupling.
[0044] In some embodiments of this disclosure, the output data generated by the target task can be evaluated through an evaluation task. In this way, after setting the evaluation criteria, the user can execute the evaluation task every time the target task generates output data, so that the target task can be evaluated by an artificial intelligence model without having to manually evaluate the target task. This improves evaluation efficiency, reduces evaluation costs, and ensures that the evaluation criteria do not change, thus guaranteeing the standardization and normalization of the evaluation.
[0045] In some embodiments of this disclosure, creating an evaluation task includes: determining a first prompt word; wherein the first prompt word is used to instruct an artificial intelligence program to generate an evaluation result; the first prompt word includes: a placeholder for the associated data of the target task, and a placeholder for the evaluation basis or the evaluation basis.
[0046] In some embodiments, the AI program is able to respond to input natural language, for example, by interacting with the AI program through text conversation. During the evaluation task, evaluation is performed by transmitting an instantiated natural language statement of a first prompt word to the AI model. The first prompt word is natural language text containing some pre-written preset text, and in addition to the preset text, placeholders, such as... Figure 3 As shown, Figure 3 The content within the box below "Evaluation Task" is the first prompt word. This first prompt word can be set when creating an evaluation task. Since the input data and corresponding output data used in each run of the target task may differ, related data is represented by placeholders. There can be one or more placeholders for related data, allowing different data within the target task's related data to be represented by multiple placeholders. For example... Figure 3In the evaluation task, `input` serves as a placeholder representing the second statement of the target task, and `output` serves as a placeholder representing the output statement of the target task. Evaluation criteria can be written into the preset text in the first prompt, or they can be represented as placeholders. Thus, when creating an evaluation task, part of the content of the first prompt (i.e., the preset text) is fixed, while another part is non-fixed content represented by placeholders. Before or during the execution of the evaluation task, the specific values of the placeholders need to be passed in. The placeholders corresponding to the evaluation criteria can be links, whose specific values can be obtained from the links when the evaluation task is executed. For placeholders for the associated data of the target task, they can be variables, whose specific values need to be input externally. For example, the evaluation task can be represented as an interface, which is run to execute the evaluation task. The interface parameters include variables corresponding to the associated data of the target task, and the specific values of these variables need to be input when the interface is executed.
[0047] In some embodiments of this disclosure, the first prompt word may further include: the role played by the artificial intelligence program, and / or, a description of the evaluation result.
[0048] In some embodiments, such as Figure 3 As shown, the content other than "evaluation task" is the first prompt word. The preset text of the first prompt word can describe the role of artificial intelligence in generating evaluation results, for example... Figure 3 The role of statements in Chinese artificial intelligence programs Figure 3 The first prompt describes the role and requires specifying the exact role, such as "You act as a teacher" or similar statements. The evaluation criteria can be included in the first prompt. Figure 3 The evaluation criteria in the document, or expressed in the form of placeholders ( Figure 3 The double brackets "{{…}}" contain placeholders, such as ref and criteria. The preset text of the first prompt can also include a description of the evaluation results, such as the format of the results and the composition of the results. Figure 3 The evaluation results description includes the strengths and weaknesses of the answer, the thought process behind the response, and the score. The first prompt word allows the AI program to fully understand its task—how to conduct the evaluation and what the expected results should be. Standardizing the format and composition of the evaluation results facilitates processing, and ensuring consistent formatting after multiple executions facilitates statistical analysis.
[0049] In some embodiments of this disclosure, the associated data of the target task is input into the evaluation task to generate an evaluation result based on the evaluation criteria through an artificial intelligence model. This includes: replacing the placeholder in the first prompt word with the associated data of the target task, or with the associated data of the target task and the evaluation criteria, to obtain a first statement, and inputting the first statement into the artificial intelligence model to obtain the evaluation result output by the artificial intelligence model.
[0050] In some embodiments, when performing the evaluation task, the associated data of the target task is input into the evaluation task to replace the placeholders of the associated data in the first prompt. If the evaluation criteria are also written in the form of placeholders in the first prompt, the specific values of the evaluation criteria are also obtained and replace the corresponding placeholders in the first prompt. In this way, the first prompt is instantiated, becoming complete natural language, i.e., the first statement, which is then input into the artificial intelligence model, enabling the artificial intelligence model to respond to the input first statement and generate the corresponding evaluation results.
[0051] In some embodiments of this disclosure, inputting the associated data of the target task into the evaluation task includes: inputting a second statement including the input data and the output data into the evaluation task; the second statement is used to describe the processing performed by the target task on the input data.
[0052] In some embodiments, the second statement is represented by a placeholder in the first prompt of the target task, such as... Figure 3 The input shown represents a placeholder for the second statement, which includes the input data. In some embodiments, the second statement is obtained by replacing the placeholder in the second prompt with the input data. Similar to the first prompt, the second prompt is a prompt for the target task. The second prompt is a statement used to tell the AI program how to achieve the function of the target task. The second prompt contains placeholders corresponding to the input data, and the second statement is obtained by instantiating the second prompt with the input data. For example, the second prompt could be: "Give a summary of {{article}} based on the input {{article}}". Here, "article" refers to a placeholder for the specific article. The second statement is passed to the evaluation task, so that the AI model can understand the function that the target task needs to achieve, the input data for this execution of the target task, and the output data obtained by achieving its function. Then, the AI model evaluates the performance of the target task based on the evaluation criteria to obtain the evaluation result.
[0053] In some embodiments of this disclosure, the evaluation criteria include: evaluation standards; or, the evaluation criteria include: evaluation standards and expected output data. In some embodiments, the evaluation standards may be, for example, specific evaluation rules, such as the evaluation standard being... Figure 3The scoring criteria can be defined as: what information in the answer earns points, and what information is omitted deducts points. Expected output data can be the standard answer; the target task outputs data based on the input data. Expected output data can be the output data the target task would ideally produce based on the input data, which is the output data the user expects. The evaluation criteria can score the output data, and the expected output data can determine the difference between the actual output data and the expected output data.
[0054] In some embodiments of this disclosure, the evaluation task is set outside the artificial intelligence model, and the evaluation task generates the evaluation result by calling the artificial intelligence model. In some embodiments, the evaluation task and the artificial intelligence program can reside in different data layers. The artificial intelligence program can be located below the evaluation task, and the evaluation task can be located above the artificial intelligence program. Setting them in different layers ensures that the evaluation task is not coupled with the artificial intelligence model, but is set outside the artificial intelligence model. The code of the evaluation task is not written into the artificial intelligence model, which reduces the coupling and facilitates modification. The evaluation task can be to input the instantiated statement of the first prompt word into the artificial intelligence model and obtain the evaluation result generated by the artificial intelligence model as the output of the evaluation task.
[0055] In some embodiments of this disclosure, the target task is a task implemented using the artificial intelligence model, or the target task is a task implemented using another artificial intelligence model. In some embodiments, when executing the target task, an artificial intelligence model may be invoked to implement the function of the target task. For example, if the target task generates an article summary based on an input article, it can be that the article is input into an artificial intelligence model to obtain the article summary. The artificial intelligence model invoked by the target task and the artificial intelligence model invoked by the evaluation task can be the same or different. The target task itself can also be an artificial intelligence model, or the target task can invoke an artificial intelligence model other than itself.
[0056] In some embodiments of this disclosure, the method further includes adjusting the parameters of the target task based on the evaluation results. In some embodiments, after obtaining the evaluation results, the parameters of the target task can be adjusted so that the subsequent output data of the target task is the same as or similar to the expected output data. In some embodiments, by repeatedly executing the method proposed in this disclosure, multiple output data of the target task can be evaluated to obtain evaluation results, thereby helping to adjust the parameters of the target task. The parameters of the target task can be, for example, the artificial intelligence program in the target task, or other parameters in the target task.
[0057] In some embodiments of this disclosure, a specific example is given below. Users can first determine the specific requirements of the target task and the expected output results. Then, a corresponding evaluation task is created based on these requirements. The construction of the evaluation task involves: selecting appropriate evaluation criteria, the expected output data, and sending the output data obtained from the target task to the evaluation task for evaluation. The evaluation task analyzes and scores the output data. Based on the scoring results, the target task can be further adjusted and optimized, and then re-evaluated, repeating this process until a satisfactory result is achieved. This method no longer relies on subjective human judgment but instead utilizes the intelligence and algorithms of artificial intelligence models for evaluation. Because the evaluation is based on unified algorithms and standards, the consistency and reliability of the results are significantly improved. It avoids differences and uncertainties caused by subjective factors in the evaluation, providing a more scientific and reliable basis for model optimization and improvement.
[0058] This disclosure also proposes an evaluation device, comprising:
[0059] Create a unit to create evaluation tasks;
[0060] The creation unit is also used to establish the association between the evaluation task and the target task to be evaluated;
[0061] A control unit is configured to respond to an event that triggers the evaluation task by inputting the associated data of the target task into the evaluation task, so as to generate evaluation results based on the evaluation criteria through an artificial intelligence model.
[0062] The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
[0063] In some embodiments, the creation of the evaluation task includes:
[0064] Determine the first prompt word; wherein the first prompt word is used to instruct the artificial intelligence program to generate an evaluation result;
[0065] The first prompt includes: placeholders for the associated data of the target task, and placeholders for the evaluation criteria or indicators of the evaluation criteria.
[0066] In some embodiments, the first prompt word may also include: a description of the role played by the artificial intelligence program and / or the evaluation results.
[0067] In some embodiments, the associated data of the target task is input into the evaluation task to generate evaluation results based on evaluation criteria using an artificial intelligence model, including:
[0068] The first statement is obtained by replacing the placeholder in the first prompt word with the associated data of the target task, or by using the associated data of the target task and the evaluation criteria. The first statement is then input into the artificial intelligence model to obtain the evaluation result output by the artificial intelligence model.
[0069] In some embodiments, inputting the associated data of the target task into the evaluation task includes: inputting a second statement including the input data and the output data into the evaluation task;
[0070] The second statement describes the processing performed by the target task on the input data.
[0071] In some embodiments, the evaluation criteria include: evaluation standards and expected output data.
[0072] In some embodiments, the evaluation task generates the evaluation results by invoking the artificial intelligence model.
[0073] In some embodiments, the target task is a task implemented using the artificial intelligence model, or the target task is a task implemented using another artificial intelligence model.
[0074] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0075] The methods and apparatus of this disclosure have been described above based on embodiments and application examples. Furthermore, this disclosure also provides an electronic device and a computer-readable storage medium, which are described below.
[0076] The following is for reference. Figure 4 The figure illustrates a structural schematic of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in the figure is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.
[0077] Electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from storage device 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.
[0078] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although an electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0079] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.
[0080] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0081] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0082] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0083] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.
[0084] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0086] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.
[0087] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0088] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0089] According to one or more embodiments of this disclosure, an evaluation method is provided, comprising:
[0090] Create an evaluation task;
[0091] Establish the association between the evaluation task and the target task to be evaluated;
[0092] In response to an event that the evaluation task is executed, the associated data of the target task is input into the evaluation task so that an evaluation result is generated by an artificial intelligence model based on the evaluation criteria.
[0093] The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
[0094] According to one or more embodiments of this disclosure, an evaluation method is provided, wherein creating an evaluation task includes:
[0095] Identify the first prompt word;
[0096] The first prompt word is used to instruct the artificial intelligence program to generate evaluation results;
[0097] The first prompt includes: placeholders for the associated data of the target task, and placeholders for the evaluation criteria or indicators of the evaluation criteria.
[0098] According to one or more embodiments of this disclosure, an evaluation method is provided, wherein the first prompt word further includes: the role played by the artificial intelligence program and / or a description of the evaluation result.
[0099] According to one or more embodiments of this disclosure, an evaluation method is provided, which inputs the associated data of the target task into the evaluation task, so as to generate evaluation results through an artificial intelligence model based on evaluation criteria, including:
[0100] The first statement is obtained by replacing the placeholder in the first prompt word with the associated data of the target task, or by using the associated data of the target task and the evaluation criteria. The first statement is then input into the artificial intelligence model to obtain the evaluation result output by the artificial intelligence model.
[0101] According to one or more embodiments of this disclosure, an evaluation method is provided, which inputs the associated data of the target task into the evaluation task, including: inputting a second statement including the input data and the output data into the evaluation task;
[0102] The second statement describes the processing performed by the target task on the input data.
[0103] According to one or more embodiments of this disclosure, an evaluation method is provided, wherein the evaluation basis includes: evaluation criteria and expected output data.
[0104] According to one or more embodiments of this disclosure, an evaluation method is provided, wherein the evaluation task generates the evaluation result by invoking the artificial intelligence model.
[0105] According to one or more embodiments of this disclosure, an evaluation method is provided, wherein the target task is a task implemented using the artificial intelligence model, or the target task is a task implemented using another artificial intelligence model.
[0106] According to one or more embodiments of this disclosure, an evaluation apparatus is provided, comprising:
[0107] Create a unit to create evaluation tasks;
[0108] The creation unit is also used to establish the association between the evaluation task and the target task to be evaluated;
[0109] A control unit is configured to respond to an event that triggers the evaluation task by inputting the associated data of the target task into the evaluation task, so as to generate evaluation results based on the evaluation criteria through an artificial intelligence model.
[0110] The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
[0111] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor;
[0112] The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method described in any one of the above.
[0113] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided for storing program code that, when executed by a processor, causes the processor to perform the methods described above.
[0114] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0115] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0116] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. An evaluation method, characterized in that, include: Create an evaluation task; Establish the association between the evaluation task and the target task to be evaluated; In response to an event that the evaluation task is executed, the associated data of the target task is input into the evaluation task so that an evaluation result is generated by an artificial intelligence model based on the evaluation criteria. The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
2. The method according to claim 1, characterized in that, The creation of the evaluation task includes: Identify the first prompt word; The first prompt word is used to instruct the artificial intelligence program to generate evaluation results; The first prompt includes: placeholders for the associated data of the target task, and placeholders for the evaluation criteria or indicators of the evaluation criteria.
3. The method according to claim 2, characterized in that, The first prompt also includes: a description of the role played by the artificial intelligence program and / or the evaluation results.
4. The method according to claim 2, characterized in that, The associated data of the target task is input into the evaluation task to generate evaluation results based on the evaluation criteria through an artificial intelligence model, including: The first statement is obtained by replacing the placeholder in the first prompt word with the associated data of the target task, or by using the associated data of the target task and the evaluation criteria. The first statement is then input into the artificial intelligence model to obtain the evaluation result output by the artificial intelligence model.
5. The method according to claim 1, characterized in that, Inputting the associated data of the target task into the evaluation task includes: inputting a second statement including the input data and the output data into the evaluation task; The second statement describes the processing performed by the target task on the input data.
6. The method according to claim 1, characterized in that, The evaluation criteria include: evaluation standards and expected output data.
7. The method according to claim 1, characterized in that, The evaluation task generates the evaluation results by calling the artificial intelligence model.
8. The method according to claim 1, characterized in that, The target task is a task implemented using the artificial intelligence model, or the target task is a task implemented using another artificial intelligence model.
9. An evaluation device, characterized in that, include: Create a unit to create evaluation tasks; The creation unit is also used to establish the association between the evaluation task and the target task to be evaluated; A control unit is configured to respond to an event that triggers the evaluation task by inputting the associated data of the target task into the evaluation task, so as to generate evaluation results based on the evaluation criteria through an artificial intelligence model. The associated data of the target task includes: the input data and the output data of the target task; the evaluation task is used to generate evaluation results of the data input to the evaluation task through an artificial intelligence model based on the evaluation criteria.
10. An electronic device, comprising: At least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method of any one of claims 1 to 8.
11. A computer-readable storage medium for storing program code that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 8.