Data evaluation method and apparatus

Abstract

Disclosed are a data evaluation method and apparatus, which relate to the technical field of machine learning. The method comprises: a computing device using an AI model having a semantic understanding capability to acquire, on the basis of an evaluation rule, evaluation criteria for instructions under second-type evaluation metrics, and to evaluate instruction data under the second-type evaluation metrics on the basis of the evaluation criteria. Thus, a computing device evaluates instructions in instruction data on the basis of evaluation criteria acquired by means of a model, and evaluation results of the instructions are generated under the same evaluation metrics, so that the evaluation results of the instructions are more comparable, thereby improving the accuracy of instruction evaluation.

Description

Data evaluation method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 20, 2024, with application number 202410323480.9 and invention name “A Data Evaluation Method and Device”. The entire contents of the application are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of machine learning technology, and in particular to a data evaluation method and device. Background Art

[0003] The computing device exploits the similarities between models, uses instruction data to fine-tune the model applied to one task, and applies the fine-tuned model to another task to reduce the resources required to train the model.

[0004] Generally, the quality of instruction data affects the efficiency of obtaining a fine-tuning model, as well as the accuracy of the prediction results and inference results obtained by the fine-tuning model. Before the computing device uses instruction data to fine-tune the model, an AI model (such as the pre-trained language model GPT-4) is used to evaluate the instruction data under various indicators (such as length, perplexity, naturalness, continuity, etc.) to improve the quality of the instruction data. However, the AI ​​model does not have quality assessment standards for various types of instruction data, and the AI ​​model cannot accurately evaluate the instruction data, resulting in the inability to effectively improve the quality of the instruction data. Therefore, how to accurately evaluate the quality of instruction data has become a problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides a data evaluation method and device for solving the problem that the AI ​​model cannot accurately evaluate instruction data, resulting in the inability to effectively improve the quality of instruction data.

[0006] In a first aspect, the present application provides a data evaluation method. The method can be executed by a computing device having an AI model deployed on the computing device. The AI ​​model has semantic understanding capabilities, such as a large language model (LLM) with semantic understanding capabilities. The method includes: the computing device obtains instruction data, the instruction data including multiple instructions. The computing device obtains a first evaluation result for each of the multiple instructions. The first evaluation result for the instruction includes a score for the instruction under a first type of evaluation metric. The first type of evaluation metric is used to describe the attributes of the instruction. The computing device uses the AI ​​model, based on the instruction data and evaluation rules, to obtain a second evaluation result for each of the multiple instructions. The second evaluation result for the instruction includes a score for the instruction under a second type of evaluation metric. The second type of evaluation metric is used to describe the characteristics of the instruction. The evaluation rules are used to indicate the degree of superiority or inferiority of different instructions under the second type of evaluation metric. The computing device obtains a score for the instruction based on the first and second evaluation results.

[0007] In this application, a computing device utilizes an AI model with semantic understanding capabilities to obtain, based on evaluation rules, the evaluation criteria for instructions under the second category of evaluation indicators, and evaluates instruction data under the second category of evaluation indicators based on the evaluation criteria. In this way, the computing device evaluates each instruction in the instruction data based on the evaluation criteria obtained by the model. The evaluation results of each instruction are generated under the same evaluation indicators, making the evaluation results of each instruction more comparable and improving the accuracy of the instruction evaluation.

[0008] In one possible implementation, a computing device utilizes an AI model to obtain a second evaluation result for each of a plurality of instructions based on instruction data and evaluation rules. This includes: the computing device displays a first prompt. The first prompt prompts a user to evaluate a portion of the plurality of instructions under a second type of evaluation metric. The computing device receives the user's evaluation results for the portion of instructions under the second type of evaluation metric based on the first prompt. The computing device utilizes the AI ​​model to obtain evaluation results for another portion of the plurality of instructions under the second type of evaluation metric based on the evaluation rules and the evaluation results for the portion of instructions under the second type of evaluation metric. The computing device obtains a second evaluation result based on the evaluation results for the portion of instructions under the second type of evaluation metric and the evaluation results for another portion of instructions under the second type of evaluation metric. In this manner, the computing device utilizes the AI ​​model to obtain evaluation criteria based on the user's evaluation results for the portion of instructions under the second type of evaluation metric. The computing device, using the evaluation criteria, obtains evaluation results for the other portion of instructions under the second type of evaluation metric. The computing device obtains the evaluation criteria for evaluating the other portion of instructions based on the user's evaluation results for the instructions under the second type of evaluation metric, thereby improving the accuracy of the evaluation results obtained by the computing device and addressing the issue of insignificant improvement in instruction quality.

[0009] In another possible implementation, the computing device uses an AI model to obtain the evaluation results of another part of the multiple instructions under the second type of evaluation indicators based on the evaluation rules and the evaluation results of a part of the instructions under the second type of evaluation indicators, including: the computing device uses at least two instructions in the part of the instructions to construct a comparison sample according to the evaluation rules. The comparison sample is used to indicate the degree of excellence of the at least two instructions under the second type of evaluation indicators. And the computing device uses the AI ​​model to obtain the evaluation results of another part of the multiple instructions under the second type of evaluation indicators based on the comparison sample and the evaluation results of the part of the instructions under the second type of evaluation indicators. In this way, the computing device uses the natural semantic understanding ability of the AI ​​model to parse out the evaluation criteria reflected by the degree of excellence of at least two instructions indicated by the comparison sample under the second type of evaluation indicators, thereby improving the accuracy of evaluating instructions under the second type of evaluation indicators.

[0010] In another possible implementation, the computing device may use different methods to obtain the evaluation results of the instruction to be evaluated under the second type of evaluation metrics, depending on the difference in the expected performance of the instruction to be evaluated under the second type of evaluation metrics and the evaluation results of each instruction in a portion of instructions under the second type of evaluation metrics. The instruction to be evaluated is one of the instructions in the other portion, as described below in different scenarios. In some possible examples, the computing device may use an AI model to compare the performance of the instruction to be evaluated with the performance of each instruction in the portion of instructions based on comparison examples.

[0011] Scenario 1

[0012] If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the worst evaluation result among the instructions, and the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the best evaluation result among the instructions, in this case, the computing device determines the evaluation result of the instruction to be evaluated under the second type of evaluation indicators based on the evaluation results of the part of instructions under the second type of evaluation indicators.

[0013] Scenario 2

[0014] If the expected performance of the instruction to be evaluated under the second type of evaluation metric is better than the instruction with the best evaluation result among the instructions, or if the expected performance of the instruction to be evaluated under the second type of evaluation metric is worse than the instruction with the worst evaluation result among the instructions, the computing device displays a second prompt message. The second prompt message is used to prompt the user to evaluate the instruction to be evaluated under the second type of evaluation metric.

[0015] Scenario 3

[0016] If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the best evaluation result among the part of instructions, in this case, the computing device determines the evaluation result of the instruction to be evaluated based on the evaluation result of the instruction with the best evaluation result among the part of instructions.

[0017] Scenario 4

[0018] If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the worst evaluation result among the part of instructions, in this case, the computing device determines the evaluation result of the instruction to be evaluated based on the evaluation result of the instruction with the worst evaluation result among the part of instructions.

[0019] In this way, based on the expected performance of the instruction to be evaluated under the second type of evaluation indicators and the difference in the quality of the evaluation results in some instructions, the computing device adopts different methods to obtain the evaluation results of the instruction to be evaluated under the second type of evaluation indicators, thereby improving the accuracy of the evaluation results of the instruction under the second type of evaluation indicators.

[0020] In another possible implementation, a computing device utilizes an AI model to obtain a second evaluation result for each of a plurality of instructions based on the instruction data and evaluation rules, including: the computing device selects, from the plurality of instructions included in the instruction data, instructions whose scores under a first type of evaluation metric are greater than or equal to a first score threshold, to obtain a set of instructions to be evaluated. Furthermore, the computing device utilizes the AI ​​model to obtain a second evaluation result based on the set of instructions to be evaluated and the evaluation rules. In this manner, the computing device uses the second type of evaluation metric to evaluate instructions in the instruction data with scores greater than or equal to the first score threshold, reducing the number of instructions that need to be evaluated using the second type of evaluation metric and improving the efficiency of the computing device in evaluating the plurality of instructions included in the instruction data.

[0021] In another possible implementation, a computing device selects a target instruction associated with a target business from the plurality of instructions based on the score of each instruction in the plurality of instructions, and generates a model of the target business based on the target instruction. The target instruction is used to indicate an instruction whose score under the second type of evaluation metric is greater than or equal to a second score threshold, and the second score threshold is used to indicate a score threshold determined for the target business. In this way, the computing device determines the score of the selected instruction based on the target business to improve the adaptability of the selected instruction to the target business.

[0022] In another possible implementation, the computing device displays at least one of the first evaluation result, the process of obtaining the first evaluation result, the second evaluation result, and the process of obtaining the second evaluation result. This allows a user to directly access the first evaluation result and the process of obtaining the first evaluation result. Furthermore, the user can directly access the second evaluation result and the process of obtaining the second evaluation result. This improves the credibility of the first and second evaluation results.

[0023] In another possible implementation, before obtaining the instruction data, the computing device displays a third prompt. This third prompt prompts the user to enter at least one of the instruction data, evaluation rules, first-category evaluation metrics, and second-category evaluation metrics. This facilitates user input of evaluation rules, first-category evaluation metrics, and second-category evaluation metrics that are consistent with the actual application scenario, thereby improving the accuracy of the evaluation results for the instruction using the first and second-category evaluation metrics.

[0024] In a second aspect, the present application provides a data evaluation device. The device includes: a transceiver module, which is used to: obtain instruction data; the instruction data includes multiple instructions. A first evaluation module is used to: obtain a first evaluation result of each instruction in the multiple instructions. The first evaluation result of the instruction includes the score of the instruction under the first type of evaluation indicators. The first type of evaluation indicators are used to describe the attributes of the instructions. The second evaluation module is used to: use the AI ​​model to obtain the second evaluation result of each instruction in the multiple instructions based on the instruction data and the evaluation rules. The second evaluation result of the instruction includes the score of the instruction under the second type of evaluation indicators. The second type of evaluation indicators are used to describe the characteristics of the instructions. The evaluation rules are used to indicate: the scores of different instructions under the second type of evaluation indicators. The processing module is used to: obtain the score of the instruction based on the first evaluation result and the second evaluation result of the instruction.

[0025] In a third aspect, the present application provides a chip. The chip includes an interface circuit and a control circuit. The interface circuit is used to obtain instruction data and, in conjunction with the control circuit, implement the method of the first aspect or any optional implementation of the first aspect.

[0026] In a fourth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The memory is configured to store a set of computer instructions. When the processor, acting as an execution device in the first aspect or any possible implementation of the first aspect, executes the set of computer instructions, the operating steps of the data processing method in the first aspect or any possible implementation of the first aspect are performed.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer software instructions that, when executed in a computing device, cause the computing device to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0028] In a sixth aspect, the present application provides a computer program product. When the computer program product is executed on a computer, the computer device is caused to execute the operation steps of the method described in the first aspect or any possible implementation of the first aspect.

[0029] The beneficial effects of the second to sixth aspects above can be referred to the description of the first aspect or any implementation of the first aspect, and will not be repeated here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG1 is a schematic diagram of the architecture of a data evaluation system provided by this application;

[0031] FIG2 is a schematic diagram of the structure of a chip provided by the present application;

[0032] FIG3 is a flow chart of a data evaluation method provided by this application;

[0033] FIG4 is a schematic diagram of a process for obtaining instruction data provided by the present application;

[0034] FIG5 is a flow chart of another data evaluation method provided by the present application;

[0035] FIG6 is an example diagram of a display method of an evaluation result provided by this application;

[0036] FIG7 is a schematic diagram of the process of the first comparison method provided in this application;

[0037] FIG8 is a schematic diagram of a flow chart of a second comparison method provided in this application;

[0038] FIG9 is a flow chart of the data evaluation method provided in this application;

[0039] FIG10 is a schematic diagram of an execution timing provided by the present application;

[0040] FIG11 is a schematic diagram of the structure of the data evaluation device provided in this application. DETAILED DESCRIPTION

[0041] The following is a brief introduction to some concepts that may be involved in this application.

[0042] Large models: A machine learning model with a large number of parameters and computing resources. The main characteristic of large models is their large number of parameters, which can improve the model's generalization and performance by using large amounts of data. Large models are more complex, with deeper and more complex network structures, which can capture richer features and relationships, thereby improving the model's expressive power. Large models include dense large models. Dense large models have a majority of non-zero parameters and are typically used in natural language processing (NLP) tasks.

[0043] Large Language Models (LLMs): Leveraging their powerful computing power and sophisticated algorithms, they can effectively process massive amounts of data, providing users with efficient and accurate information processing and analysis services. LLMs excel in understanding and generating human language. For example, large language models are widely used in automated question-answering systems, text summarization, machine translation, and language generation. Furthermore, the self-learning capabilities of large language models enable them to continuously evolve, continuously improving their performance and intelligence through continuous learning from new data.

[0044] It is worth noting that in some optional situations, the large language model can also be referred to as a large model. The large model provided in the embodiment of the present application can refer not only to a large language model, but also to a model whose model parameters reach a certain level. For example, according to the changes in the field in which the model is applied, the large model can also refer to a model with various functions such as image processing functions, human-computer interaction functions, semantic search and dialogue functions. This application does not limit the fields in which the large model can be applied and the specific name. In this article, for the sake of simplicity of description, it is named as a large model, but this should not be understood as a limitation of this application and will not be repeated later.

[0045] Method 1: manually evaluate multiple instructions included in the initial instruction data.

[0046] Based on their professional knowledge, the staff member evaluates the multiple instructions included in the initial instruction data for training the model one by one, and obtains an evaluation result for each of the multiple instructions. The staff member then selects multiple instructions whose evaluation results meet the requirements from the multiple instructions, thereby obtaining the instruction data for training the model.

[0047] Under this approach, staff members individually evaluate and screen the multiple instructions included in the initial instruction data, which is time-consuming and labor-intensive. Furthermore, when multiple staff members evaluate and screen the initial instruction data, due to differences in professional knowledge among staff members, they may use different evaluation criteria to evaluate instructions and different screening criteria to screen instructions, thus failing to ensure the quality of the screened instructions.

[0048] In method 2, the computing device uses a large model to evaluate multiple instructions included in the initial instruction data.

[0049] The computing device uses a large model to select instructions that meet the similarity requirements from the multiple instructions included in the initial instruction data based on the similarity of the multiple instructions included in the initial instruction data, and obtains instruction data for training the model.

[0050] Under this method, the large model lacks appropriate evaluation criteria, resulting in inaccurate evaluation results of instructions, and the quality of the screened instructions cannot be guaranteed.

[0051] As can be seen from the above, the quality of instructions obtained by the methods usually used to improve the quality of instructions cannot be guaranteed. Based on this, the present application provides a data evaluation method, in which a computing device obtains instruction data. The instruction data includes multiple instructions. The computing device obtains a first evaluation result of each instruction in the multiple instructions. The first evaluation result of the instruction includes the score of the instruction under the first type of evaluation indicator. The first type of evaluation indicator is used to describe the attributes of the instruction. The computing device uses an AI model to obtain a second evaluation result of the instruction based on the instruction data and the evaluation rules. The second evaluation result of the instruction includes the score of the instruction under the second type of evaluation indicator. The second type of evaluation indicator is used to describe the characteristics of the instruction. The evaluation rules are used to indicate: the degree of pros and cons of different instructions under the second type of evaluation indicator. The computing device obtains the score of the instruction based on the first evaluation result and the second evaluation result of the instruction.

[0052] In this way, the computing device uses an AI model with semantic understanding capabilities to obtain the evaluation criteria for the instruction under the second type of evaluation indicators based on the evaluation rules, and then evaluates the instruction data based on the second type of evaluation indicators based on the evaluation criteria. In this way, the computing device evaluates each instruction in the instruction data based on the evaluation criteria obtained by the model. The evaluation results of each instruction are generated under the same evaluation indicators, making the evaluation results of each instruction more comparable, thereby improving the accuracy of the instruction evaluation.

[0053] This application can be applied not only to existing machine learning technology, artificial intelligence (AI) technology and optimization scenarios of data evaluation, but also to future machine learning technology, AI technology and optimization scenarios of data evaluation. The terms used in the implementation method part of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0054] The following is an illustrative description of the scenarios in which the embodiments of the present application can be applied, with reference to the accompanying drawings.

[0055] FIG1 is a schematic diagram of the architecture of a data evaluation system provided by this application. As shown in FIG1 , the data evaluation system includes a computing device 110. Computing device 110 can be an electronic device with computing capabilities or a virtual device with computing capabilities. If computing device 110 is an electronic device with computing capabilities, computing device 110 can be a server, a personal computer, a tablet computer, etc. If computing device 110 is a virtual device with computing capabilities, computing device 110 can be a virtual machine, a container, etc.

[0056] The user may input instruction data to be evaluated into the computing device 110 , and the computing device 110 evaluates the multiple instructions included in the instruction data, and the computing device 110 outputs evaluation results of the instructions.

[0057] Computing device 110 includes an input / output (I / O) interface 114, a processor 111, and a memory 112. I / O interface 114 is used to communicate with devices external to computing device 110. For example, instruction data to be evaluated, evaluation rules, first-category evaluation indicators, and second-category evaluation indicators are received from computing device 110 via I / O interface 114. Computing device 110 processes the input data (e.g., evaluates the instruction data) and then outputs the processing results via I / O interface 114.

[0058] The processor 111 is the computing core and control core of the computing device 110. It may include: a central processing unit (CPU), a specific integrated circuit, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. In actual applications, the computing device 110 may also include multiple processors. The processor 111 may include one or more processor cores. An operating system and other software programs are installed in the processor 111, so that the processor 111 can access the memory 112 and various peripheral component interconnect (Peripheral Component Interconnect e9press, PCIe) devices.

[0059] Processor 111 is connected to memory 112 via bus 116. Bus 116 can be a double data rate (DDR) bus or other types of buses. Memory 112 is the main memory of computing device 110. Memory 112 is typically used to store various running software in the operating system, received instruction data, evaluation rules, first-category evaluation indicators and second-category evaluation indicators, and processing results to be output. To improve the access speed of processor 111, memory 112 needs to have the advantage of fast access speed. In traditional computer devices, dynamic random access memory (DRAM) is typically used as memory 112. In addition to DRAM, memory 112 can also be other random access memories, such as static random access memory (SRAM). Memory 112 can also be read-only memory (ROM). For example, read-only memory can be programmable read-only memory (PROM) or erasable programmable read-only memory (EPROM). This embodiment does not limit the quantity and type of the memory 112 .

[0060] Optionally, to store data persistently, the data evaluation system is further provided with a data storage system 113. The data storage system 113 may be located external to the computing device 110 (as shown in FIG1 ) and exchange data with the computing device 110 via a network. Alternatively, the data storage system 113 may be located internal to the host computer, for example, where the data storage system 113 exchanges data with the processor 111 via a bus 116. In this case, the data storage system 113 is implemented as a hard disk.

[0061] Optionally, the data evaluation system may further include a client device 120. A user may input command data to be evaluated into the computing device 110 via the client device 120, and the computing device 110 may send processing results to the user via the client device 120. The client device 120 is a terminal device, including but not limited to a personal computer, server, mobile phone, tablet computer, or smart car.

[0062] Exemplarily, the processor 111 in Figure 1 can be implemented by a chip, as shown in Figure 2, which is a structural diagram of a chip provided by this application. Exemplarily, the chip 200 includes a core 201, a CPU 202, a system buffer 203 and a DDR 206.

[0063] Among them, CPU 202 is used to accept AI tasks (such as instruction evaluation tasks) and call core 201 to execute the tasks. When chip 200 has multiple cores 201, CPU 202 is also used to take on scheduling tasks. For example, CPU 202 can be implemented by an ARM processor, which is small in size, low in power consumption, uses a 32-bit reduced instruction set, and has simple and flexible addressing. Of course, in some embodiments, CPU 202 can also be implemented by other processors.

[0064] Core 201 is used to provide the computing power required for instruction evaluation tasks. In an optional scenario, core 201 includes a load / store unit (LSU), a cube computing unit, a scalar computing unit, a vector computing unit, and a buffer. Among them, the LSU is used to load data to be evaluated and store evaluation results. It can also be used for reading and writing internal data between different buffers in the core, and to complete some format conversion operations. The cube computing unit is used to provide the core computing power for matrix multiplication. The scalar computing unit is a single instruction single data (SISD) processor. This type of processor only processes one piece of data (usually an integer or floating point number) at the same time. The vector computing unit, also known as an array processor, is a processor that can directly operate a group of arrays or vectors for calculation. The number of buffers may be one or more. For example, the buffer mainly refers to the level 1 buffer (L1 buffer). The buffer is used to temporarily store some data that the core 201 needs to use repeatedly to reduce reading and writing from the bus. In addition, the implementation of certain data format conversion functions also requires that the source data is located in the buffer. In this embodiment, since the buffer is located in the core, the distance between the cube computing unit in the core and the storage area where the data is located is shortened, reducing the cube computing unit's access to DDR 206, thereby reducing data access latency and core data processing latency.

[0065] The system buffer 203 mainly refers to a level 2 buffer (L1 buffer or L2 cache), which is used to temporarily store input data, intermediate results or final results passing through the chip.

[0066] DDR 206 is an off-chip memory that can be replaced with high bandwidth memory (HBM) or other off-chip memory. DDR 206 is located between the chip and the external memory, overcoming the access speed limitations of shared memory reads and writes for computing resources.

[0067] The input / output (I / O) device 205 included in chip 200 refers to the hardware that performs data transmission and can also be understood as the device that interfaces with the I / O interface. Common I / O devices include network cards, printers, keyboards, and mice. All external storage devices, such as hard drives, floppy disks, and optical disks, can also serve as I / O devices.

[0068] In some application scenarios, data encoding or decoding is required. Therefore, chip 200 may also include a codec 204 and an I / O device 205. Codec 204 is used to encode or decode data. It should be understood that in some optional scenarios, codec 204 may also be designed as a codec unit (software module) and integrated into core 201.

[0069] The core 201, CPU 202, system buffer 203, codec 204, I / O device 205, and DDR 206 are connected via a bus. The bus may include a path for transmitting information between the above components (such as CPU 202 and system buffer 203). In addition to the data bus, the bus may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus may be a PCIe bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (C9L), a cache coherent interconnect for accelerators (CCI9), etc. For example, the core 201 can access these I / O devices 205 via the PCIe bus. The core 201 is connected to the system buffer 203 via the DDR bus. Here, different system buffers 203 may use different data buses to communicate with the core 201. Therefore, the DDR bus can also be replaced with other types of data buses. The embodiment of the present application does not limit the bus type.

[0070] For example, after CPU 202 loads the data to be processed by the AI ​​task (e.g., instruction data to be evaluated) into DDR 206, the LSU in core 201 reads (loads) the data from DDR 206 and evaluates the data to obtain a processing result (i.e., an evaluation result). After obtaining the processing result, the LSU then loads (stores) the processing result into DDR 206. The network interface card then sends the evaluation result to client device 120 or to data storage system 113 for persistent storage.

[0071] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the computing device. In other embodiments, the computing device and chip may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0072] In an embodiment of the present application, an AI model may be deployed in a computing device or a processor (or chip) in the computing device. The AI ​​model has semantic understanding capabilities. The AI ​​model may be a large language model (LLM), or other neural network model or algorithm model with semantic understanding capabilities, etc., which is not limited in this application.

[0073] The data evaluation method provided in this application is described in detail below in conjunction with the contents shown in Figures 1 and 2.

[0074] Figure 3 is a flow chart of a data evaluation method provided by the present application. The data evaluation method can be executed by a computing device or a chip or processor in a computing device. The data evaluation method can be executed by a computing device, which can be the computing device 110 shown in Figure 1, or the chip shown in Figure 2, etc. Regarding the hardware implementation of the computing device, please refer to the description of Figures 1 and 2 above, which will not be described in detail here. In some optional examples, the data evaluation method can also be executed by other computing devices. Regarding the hardware implementation of other computing devices, please refer to the description of Figures 1 and 2 above, which will not be described in detail here. Here, the data evaluation method provided by the present application is explained by taking the computing device 110 executing the data evaluation method provided by this embodiment as an example. As shown in Figure 3, the data evaluation method provided by this embodiment includes the following S310 to S340.

[0075] S310, the computing device obtains instruction data.

[0076] The instruction data includes a plurality of instructions, which can be used to train a model so that the trained model can be applied to a target business scenario.

[0077] Optionally, the instruction types of the instructions included in the instruction data may be different depending on the target business scenario to which the model is applied.

[0078] For example, for a scenario involving language processing (such as a natural language processing scenario), the instruction type of the instructions included in the instruction data may be text.

[0079] For another example, for a scenario involving images (such as an image recognition scenario), the instruction type of the instructions included in the instruction data may be a picture.

[0080] For another example, for a scenario involving voice (such as a language recognition scenario), the instruction type of the instructions included in the instruction data may be audio.

[0081] For another example, for a scene involving video (such as a scene of animation recognition), the instruction type of the instruction included in the instruction data may be video.

[0082] For example, for a multimodal scenario, the instruction data may include instructions of various types, such as pictures, text, audio, video, and the like.

[0083] The following describes the data evaluation method provided in the embodiments of the present application, taking the natural language processing scenario as an example. For situations where the target business scenario for the model is other scenarios, the same processing method as for the natural language processing scenario can also be used to process the instruction data of the model applied to other scenarios to improve the quality of the training data used to train the model. This application does not elaborate on this.

[0084] The computing device may obtain the multiple instructions included in the instruction data in different ways. Three possible ways are given below.

[0085] In a first possible manner, the computing device directly receives multiple instructions input by the user, and uses the multiple instructions as instruction data.

[0086] In a second possible manner, the computing device receives multiple instructions sent by the terminal device, and obtains instruction data according to the multiple instructions.

[0087] In this approach, a computing device can receive multiple texts sent by one or more terminal devices, obtain multiple target texts based on these multiple texts, and use these multiple target texts as instruction data. The target texts reflect the semantics, logical relationships, knowledge structure, and other content of the target business scenario.

[0088] Exemplarily, the computing device receives multiple texts of user evaluations of products sent by the terminal device, and the computing device obtains multiple target texts such as user evaluations of products, user inquiries about products, etc. based on the multiple texts, and uses the target texts as instruction data.

[0089] In a third possible approach, a computing device receives multiple reference instructions sent by a terminal device, generates multiple instructions based on the multiple reference instructions using an instruction generation model, and obtains instruction data based on the multiple reference instructions and the multiple instructions generated by the model.

[0090] FIG4 is a flow chart of obtaining instruction data provided by the present application. As shown in FIG4 , the computing device may execute the following steps ① to ③ to obtain instruction data.

[0091] ①. The computing device receives multiple reference instructions sent by one or more terminal devices, and constructs an instruction data pool according to the multiple reference instructions.

[0092] ② The computing device uses the multiple reference instructions included in the instruction data pool to fine-tune the initial instruction generation model used to generate instructions, thereby obtaining a fine-tuned instruction generation model. This initial instruction generation model can be a large language model (large model or LLM), or other neural network model or algorithm model with instruction generation capabilities, which is not limited in this application.

[0093] ③. The computing device generates multiple instructions using the fine-tuned instruction generation model.

[0094] In some possible scenarios, the computing device may also perform step ④. Step ④ specifically includes: the computing device evaluating the fine-tuned instruction generation model. For example, the computing device may evaluate multiple instructions generated by the fine-tuned instruction generation model to evaluate the fine-tuned instruction generation model. For details on how the computing device evaluates instructions, please refer to the detailed description of S320 to S340 below and will not be repeated here.

[0095] In some possible scenarios, the computing device may further perform step ⑤. Step ⑤ specifically includes: the computing device fine-tuning an initial instruction generation model for generating instructions using instructions that meet preset conditions among the multiple instructions.

[0096] In one possible scenario, before executing S310, the computing device may further execute S310A. S310A specifically includes: the computing device displaying a third prompt message. The third prompt message is used to prompt the user to input at least one of the following: instruction data, evaluation rules, first-category evaluation indicators, and second-category evaluation indicators.

[0097] The first category of evaluation metrics is used to describe the attributes of an instruction. Instruction attributes indicate properties that can be objectively and unequivocally determined based on the instruction. This means that the evaluation results of the first category of evaluation metrics are negligibly affected by the staff. The first category of evaluation metrics may include one or more evaluation indicators. These indicators may include length, perplexity, and so on. In some possible examples, instruction attributes may also be referred to as inherent attributes of the instruction, instruction metadata, and so on. Evaluation indicators may also be referred to as evaluation dimensions. The first category of evaluation metrics may also be referred to as absolute standards, specific dimensions, and so on.

[0098] The second category of evaluation indicators describes the characteristics of instructions. These characteristics indicate properties that cannot be objectively and unequivocally determined from the instructions. This means that the evaluation results of these second-category evaluation indicators are subject to significant influence from the staff. This category of evaluation indicators can include one or more indicators. These indicators might include fluency, continuity, naturalness, or criticality. These second-category evaluation indicators can also be referred to as relative standards or abstract dimensions.

[0099] Evaluation rules are used to indicate the relative performance of different instructions under the second category of evaluation metrics. Evaluation rules can be short text templates used to compare the performance of different instructions under the second category of evaluation metrics. Taking fluency, a second category of evaluation metrics, as an example, evaluation rules could be short text templates such as "Instruction 1 is more fluent than instruction 2," "Instruction 1 is better than instruction 2," "Instruction 1 is better than instruction 2," "Instruction 2 is less fluent than instruction 1," "Instruction 2 is worse than instruction 1," "Instruction 2 is worse than instruction 1," and so on.

[0100] S320: The computing device obtains a first evaluation result of each instruction in the plurality of instructions.

[0101] The first evaluation result of the instruction includes the score of the instruction under the first category of evaluation indicators.

[0102] Depending on the number of evaluation indicators included in the first category of evaluation indicators, the computing device obtains the first evaluation result of the instruction included in the instruction data under the first category of evaluation indicators in different ways, which are described below in different situations.

[0103] In case a, the first type of evaluation indicators includes one evaluation indicator.

[0104] In this case, the computing device evaluates the score of each instruction included in the instruction data under a single evaluation indicator included in the first category of evaluation indicators, and obtains a first evaluation result of the instruction under the first category of evaluation indicators based on the score of each instruction under the single evaluation indicator included in the first category of evaluation indicators.

[0105] The computing device may obtain a first evaluation result of the instruction under the first category of evaluation indicators according to the score of the instruction under a single evaluation indicator included in the first category of evaluation indicators in a variety of ways. Two possible ways are given below.

[0106] In mode a1, the computing device directly uses the score of the instruction under a single evaluation indicator included in the first category of evaluation indicators as the first evaluation result of the instruction under the first category of evaluation indicators.

[0107] Exemplarily, the instruction data includes texts 1 to n. The computing device calculates the scores of texts 1 to n under the length metric, respectively, to obtain scores 1 to n corresponding to the length metric of texts 1 to n. The computing device uses score 1 as the first evaluation result of text 1 under the first type of evaluation metric, and similarly uses score n as the first evaluation result of text n under the first type of evaluation metric.

[0108] In mode a2, the computing device processes the score of the instruction under a single evaluation metric included in the first category of evaluation metrics to obtain a first evaluation result for the instruction under the first category of evaluation metrics. The processing may include, but is not limited to, multiplication, division, addition, or subtraction of a coefficient, or one or more of the following.

[0109] Exemplarily, the instruction data includes text 1 to text n. The computing device calculates the scores of text 1 to text n under the length index respectively, and obtains scores 1 to score n corresponding to text 1 to text n under the length index respectively. The computing device multiplies, divides, adds or subtracts a coefficient 1 from the score corresponding to each text under the length index, and uses the calculation result as the first evaluation result of the text under the first type of evaluation index, such as multiplying, dividing, adding or subtracting the score 1 by the coefficient 1 to obtain the calculation result 1 as the first evaluation result of text 1 under the first type of evaluation index.

[0110] In case b, the first type of evaluation indicators includes multiple evaluation indicators.

[0111] In this case, the computing device can obtain the score of each instruction in the instruction data under each evaluation indicator included in the first category of evaluation indicators. Each evaluation indicator corresponds to one score. The process of the computing device obtaining the score of an instruction under an evaluation indicator included in the first category of evaluation indicators can use the method described in case a, and will not be repeated here.

[0112] For example, the first category of evaluation indicators includes evaluation indicators 1 through m. The instruction data includes texts 1 through n. The computing device can employ the method described in scenario a to obtain the scores of each text in the instruction data under each evaluation indicator included in the first category of evaluation indicators. If the computing device employs the method described in scenario a, scores 11 through 1m are obtained for text 1 under evaluation indicators 1 through m included in the first category of evaluation indicators.

[0113] After obtaining the scores of the instructions under various evaluation indicators, the computing device can use multiple methods to obtain the first evaluation results of the instructions under the first category of evaluation indicators based on the scores of the instructions under various evaluation indicators. Two possible methods are given below.

[0114] In mode b1, the computing device performs a weighted operation on the scores of the instruction under various evaluation indicators to obtain a first evaluation result of the instruction under a first type of evaluation indicators.

[0115] Exemplarily, the computing device performs a weighted operation on score 11 and score 1m, respectively, to obtain score 1 for text 1 under the first category of evaluation indicators. The computing device uses score 1 as first evaluation result 1 for text 1 under the first category of evaluation indicators. Similarly, the computing device uses the method for obtaining score 1 to calculate first evaluation results 2 to first evaluation results n for texts 2 to n under the first category of evaluation indicators, respectively, which will not be detailed here.

[0116] In mode b2, the computing device calculates the sum of the scores of the instruction under each evaluation metric and processes the sum to obtain a first evaluation result for the instruction under the first category of evaluation metrics. The processing may include, but is not limited to, one or more of multiplication, division, addition, and subtraction of a coefficient.

[0117] Exemplarily, the computing device performs a sum operation on the score 11 and the score 1m, and multiplies, divides, adds, or subtracts the calculated sum by a coefficient 1 to obtain a calculation result 1. The computing device uses the calculation result 1 as the first evaluation result 1 for text 1 under the first category of evaluation indicators. Similarly, the computing device uses the method for obtaining score 1 to calculate first evaluation results 2 to first evaluation results n for texts 2 to n under the first category of evaluation indicators, respectively, which will not be repeated here.

[0118] In one possible scenario, the computing device may use a model to evaluate the instructions included in the instruction data to obtain scores for the instructions under various evaluation indicators included in the first category of evaluation indicators. The model may be a lightweight neural network model, for example. Depending on the needs of the actual application, the computing device may use a single model to calculate the scores of the instructions under a single evaluation indicator, or may use a single model to calculate the scores of the instructions under multiple evaluation indicators, which is not limited in this application.

[0119] In a possible scenario, the computing device displays at least one of the first evaluation result and a process of obtaining the first evaluation result.

[0120] The process of obtaining the first evaluation result may refer to a process in which the computing device obtains the first evaluation result of the instruction under the first type of evaluation indicators.

[0121] Taking the length in the first category of evaluation indicators as an example, the process of obtaining the first evaluation result of the instruction includes: obtaining the length of the instruction, comparing the instruction length with the length standard, and obtaining the first evaluation result of the instruction. For example, if the length of the instruction is 10 bytes (Byte, B), the length standard 1 is: 12B, and the score is 5 points. The length standard 2 is: 8B, and the score is 3 points. In this case, it is determined that the score of the instruction under the length indicator is 4 points. In this case, the process of obtaining the first evaluation includes: the length of the instruction: 10B. The length of the instruction is less than the length standard 1 (the score is 5 points) and greater than the length standard 2 (the score is 3 points). It is determined that the score of the instruction under the length indicator is 4 points.

[0122] The manner in which the computing device displays the first evaluation result and the process of obtaining the first evaluation result is described below with respect to FIG6 and is not further described here. This facilitates the user's acquisition of the first evaluation result of the instruction under the first category of evaluation indicators and the process of obtaining the first evaluation result, thereby improving the credibility of the first evaluation result.

[0123] S330: The computing device uses the AI ​​model to obtain a second evaluation result of each instruction in the multiple instructions according to the instruction data and the evaluation rules.

[0124] The second evaluation result of the instruction includes the score of the instruction under the second type of evaluation indicators, which are used to describe the characteristics of the instruction. The evaluation rules are used to indicate the degree of excellence of different instructions under the second type of evaluation indicators.

[0125] In some possible scenarios, the process of a computing device using an AI model to obtain a second evaluation result for each of a plurality of instructions based on instruction data and evaluation rules may include: the computing device prompting a user to evaluate a portion of the instructions in the instruction data under a second type of evaluation metric, and obtaining the user's evaluation results for the portion of instructions under the second type of evaluation metric. Furthermore, the computing device uses the user's evaluation results for the portion of instructions under the second type of evaluation metric as a reference to obtain the evaluation results for another portion of the instructions included in the instruction data under the second type of evaluation metric. In this way, by using the user's evaluation results for the portion of instructions under the second type of evaluation metric as a reference, the computing device improves the accuracy of the computing device's evaluation of the instructions under the second type of evaluation metric. This process may specifically include the following ① to ④.

[0126] ①. The computing device displays the first prompt information.

[0127] The first prompt information is used to prompt the user to evaluate a portion of the multiple instructions under the second type of evaluation indicators.

[0128] The computing device can display the first prompt information in a variety of ways, for example, the computing device can display the first prompt information in a pop-up window, or the computing device can display the first prompt information in text format. According to the needs of actual applications, the computing device can also display the first prompt information in other ways, and this application does not limit this.

[0129] ②. The computing device receives the user's evaluation results of a portion of instructions under the second type of evaluation indicators based on the first prompt information.

[0130] The evaluation results of this part of instructions and part of instructions under the second type of evaluation indicators can be called data samples, reference instructions, etc. The computing device can use a variety of methods to receive the user's evaluation results of a part of instructions under the second type of evaluation indicators based on the first prompt information. For example, the computing device directly receives the evaluation results of a part of instructions in the instruction data under the second type of evaluation indicators input by the user through the input device. For another example, the computing device receives the evaluation results of a part of instructions in the instruction data under the second type of evaluation indicators transmitted by the user to the computing device through the client. According to the needs of actual applications, the computing device can also use other methods to receive the user's evaluation results of a part of instructions under the second type of evaluation indicators, and this application is not limited to this.

[0131] The user evaluates each instruction in a portion of instructions using each evaluation metric included in the second category of evaluation metrics, obtaining an evaluation result for each instruction. Users can evaluate instructions in various ways. The following example illustrates how users can evaluate instructions, using the example of instruction data including text 1 through text n and a portion of instructions including text 1 through text i.

[0132] In case 1, the user evaluates a portion of instructions one at a time.

[0133] In this case, the user evaluates one instruction in a portion of instructions at a time and obtains an evaluation result of the instruction under each evaluation indicator included in the second type of evaluation indicators. Table 1 is a possible example of the evaluation result of the instruction obtained by the user.

[0134] Table 1: The first possible example

[0135] In case 2, the user evaluates one or more instructions in a portion of instructions at a time.

[0136] In this case, the user evaluates one or more instructions in a portion of the instructions each time and obtains evaluation results of the one or more instructions under each evaluation metric included in the second category of evaluation metrics. Table 2 is another possible example of the evaluation results of the instructions obtained by the user.

[0137] Table 2 Second possible example

[0138] Tables 1 and 2 above show the user's evaluation results for each instruction in a portion of instructions under the second category of evaluation indicators, when the second category of evaluation indicators includes one evaluation indicator. In some possible examples, the second category of evaluation indicators may also include multiple evaluation indicators. In this case, the evaluation results for each instruction under each evaluation indicator can be obtained using the method described above, which is not detailed here.

[0139] ③. The computing device uses the AI ​​model to obtain the evaluation results of another part of the multiple instructions under the second type of evaluation indicators based on the evaluation rules and the evaluation results of a part of the instructions under the second type of evaluation indicators. This process may include the following steps (1) and (2).

[0140] In step (1), the computing device uses at least two instructions from a portion of instructions to construct a comparison sample according to the evaluation rule.

[0141] The comparison sample is used to indicate the degree of superiority or inferiority of at least two instructions under the second type of evaluation indicators. The computing device selects, from a portion of the instructions evaluated by the user, a number of instructions equal to the number of instructions involved in the short text template according to the number of instructions involved in the short text template shown in the evaluation rule, and uses the selected instructions in the form of the short text template shown in the evaluation rule to construct a comparison sample. According to the needs of the actual application, the computing device can also adopt the method of constructing comparison samples described above, using a portion of instructions to construct multiple comparison samples according to the evaluation rules.

[0142] The following uses an example in which a computing device uses at least two instructions from a portion of instructions and the short text template shown in the evaluation rule is "Instruction 1 is better than Instruction 2" to illustrate the process of the computing device constructing a comparison sample according to the evaluation rule.

[0143] The computing device selects any two texts x1 and y1 with different scores from texts 1 to i. If the score of text x1 is higher than the score of text y1, the computing device constructs a comparison example according to the "Instruction 1 is better than Instruction 2" method, resulting in comparison example 1: "Text x1 is better than text y1."

[0144] In the case where two comparison examples need to be constructed, the computing device can continue to select any two texts with different scores from text 1 to text i that do not include text x1 and text y1, such as x2 and text y2. If the score of text x2 is higher than the score of text y2, in this case, the computing device continues to construct the comparison example in the manner of "instruction 1 is better than instruction 2" to obtain comparison example 2 "text x2 is better than text y2". According to the needs of actual applications, in the case where more comparison examples need to be constructed, the computing device can continue to construct more comparison examples by referring to the method described above, and this application does not elaborate on this.

[0145] In step (2), the computing device uses the AI ​​model to obtain the evaluation results of another part of the multiple instructions under the second type of evaluation indicators based on the comparison samples and the evaluation results of a part of the instructions under the second type of evaluation indicators.

[0146] The degree of excellence of at least two instructions indicated by the comparison sample under the second type of evaluation indicators reflects the standard for judging the excellence of the instructions. The computing device uses the AI ​​model to obtain the standard for judging the excellence of the instructions reflected by the comparison sample. The computing device compares the instruction to be evaluated with each instruction in a part of instructions, and obtains the degree of excellence of the instruction to be evaluated and each instruction in a part of instructions based on the standard. The instruction to be evaluated is an instruction in another part of instructions. The computing device can use a variety of methods to compare the instruction to be evaluated with each instruction in a part of instructions, such as binary search method, sequential search method, etc. Regarding the process of the computing device using the binary search method to compare the instruction to be evaluated with each instruction in a part of instructions, please refer to the relevant description of Figure 7 below. Regarding the process of the computing device using the sequential search method to compare the instruction to be evaluated with each instruction in a part of instructions, please refer to the relevant description of Figure 8 below, which will not be repeated here. In this application, taking the example of a computing device using binary search and sequential search methods to compare the instruction to be evaluated with each instruction in a part of instructions, the process of the computing device using the comparison of the quality of the instruction to be evaluated and each instruction in a part of instructions is shown. According to the needs of actual applications, the computing device can also use other algorithms to compare the quality of the instruction to be evaluated and each instruction in a part of instructions, such as interpolation search method, Fibonacci search method, tree table search method, block search method, hash search method, etc., and this application does not limit this.

[0147] Depending on the different levels of quality of the instruction to be evaluated and each instruction in a portion of instructions, the computing device may adopt different methods to obtain the evaluation results of the instruction to be evaluated under the second type of evaluation indicators, which are described below in different situations.

[0148] In case A, the expected performance of the instruction to be evaluated under the second type of evaluation indicators is within the evaluation results of a part of the instructions under the second type of evaluation indicators.

[0149] The expected performance of the evaluation instruction under the second type of evaluation indicators may refer to the expected score of the instruction to be evaluated under the second type of evaluation indicators.

[0150] If the expected performance of the instruction to be evaluated under the second type of evaluation metric is better than the instruction with the worst evaluation result among the instructions, and if the expected performance of the instruction to be evaluated under the second type of evaluation metric is worse than the instruction with the best evaluation result among the instructions, in this case, the computing device determines the evaluation result of the instruction to be evaluated under the second type of evaluation metric based on the evaluation results of the instructions under the second type of evaluation metric. For details about this process, please refer to the relevant descriptions in Figures 7 and 8 below and will not be repeated here.

[0151] For example, the second evaluation metric is fluency. A portion of instructions includes instruction 1 and instruction 2. Instruction 1 is the instruction with the highest fluency score among the portion of instructions, and instruction 2 is the instruction with the lowest fluency score among the portion of instructions. If the computing device uses the AI ​​model to determine that the fluency of the instruction to be evaluated is worse than that of instruction 1 and better than that of instruction 2, the computing device determines the fluency score of the instruction to be evaluated based on the fluency scores of the portion of instructions.

[0152] The following example illustrates how a computing device can use an AI model to compare the fluency of an instruction to be evaluated with instructions 1 and 2.

[0153] For example, instruction 1 is "It is sunny today." Instruction 2 is "It is sunny today." The instruction to be evaluated is "It is sunny today." The computing device can use the AI ​​model to determine that the fluency of the instruction to be evaluated is better than that of instruction 2, and the fluency of the instruction to be evaluated is worse than that of instruction 1.

[0154] For another example, instruction 1 is "cold." Instruction 2 is "it's cold today." The instruction to be evaluated is "the weather is cold today." The computing device can use the AI ​​model to determine that the fluency of the instruction to be evaluated is better than that of instruction 1, and that the fluency of the instruction to be evaluated is worse than that of instruction 2.

[0155] In case B, the expected performance of the instruction to be evaluated under the second type of evaluation indicators is outside the evaluation results of some instructions under the second type of evaluation indicators.

[0156] In this case, the computing device may adopt two possible methods to obtain the evaluation result of the instruction to be evaluated under the second type of evaluation indicators.

[0157] Method 1

[0158] If the expected performance of the instruction to be evaluated under the second type of evaluation metric is better than the instruction with the best evaluation result among the instructions, or if the expected performance of the instruction to be evaluated under the second type of evaluation metric is worse than the instruction with the worst evaluation result among the instructions, the computing device displays a second prompt message. The second prompt message is used to prompt the user to evaluate the instruction to be evaluated under the second type of evaluation metric. The computing device then receives the evaluation result of the instruction to be evaluated under the second type of evaluation metric, sent by the user based on the second prompt message.

[0159] Exemplarily, the second type of evaluation indicator is fluency. A portion of instructions includes instruction 1 and instruction 2, instruction 1 is the instruction with the highest score in terms of fluency among the portion of instructions, and instruction 2 is the instruction with the lowest score in terms of fluency among the portion of instructions. If the computing device uses the AI ​​model to determine that the fluency of the instruction to be evaluated is better than instruction 1, or worse than instruction 2. In this case, the computing device displays a second prompt message, prompting the user to evaluate the fluency of the instruction to be evaluated. And receive the user's score for the instruction to be evaluated in terms of fluency. Regarding the process of the computing device determining the pros and cons of the instruction in terms of fluency, please refer to the relevant description above and will not be repeated here.

[0160] Method 2

[0161] If the expected performance of the instruction to be evaluated under the second type of evaluation indicator is better than the instruction with the best evaluation result among the instructions, in this case, the computing device determines the evaluation result of the instruction to be evaluated based on the evaluation result of the instruction with the best evaluation result among the instructions. Alternatively, if the expected performance of the instruction to be evaluated under the second type of evaluation indicator is worse than the instruction with the worst evaluation result among the instructions, the computing device determines the evaluation result of the instruction to be evaluated based on the evaluation result of the instruction with the worst evaluation result among the instructions.

[0162] Exemplarily, the second type of evaluation indicator is fluency. A portion of instructions includes instruction 1 and instruction 2, where instruction 1 is the instruction with the highest fluency score among the portion of instructions, and instruction 2 is the instruction with the lowest fluency score among the portion of instructions. If the computing device uses the AI ​​model to determine that the fluency of the instruction to be evaluated is better than that of instruction 1, the computing device determines that the score of the instruction to be evaluated in terms of fluency is the score of instruction 1 in terms of fluency. Alternatively, if the fluency of the instruction to be evaluated is worse than that of instruction 2, the computing device determines that the score of the instruction to be evaluated in terms of fluency is the score of instruction 2 in terms of fluency.

[0163] The above description uses fluency as an example of the second category of evaluation metrics to illustrate how a computing device uses the evaluation results of a portion of instructions under the second category of evaluation metrics to obtain the evaluation results of another portion of instructions under the second category of evaluation metrics. In some possible examples, the second category of evaluation metrics may include multiple evaluation metrics. In such cases, the computing device can use the method described above to obtain the evaluation results of the other portion of instructions under each evaluation metric, which is not further described here.

[0164] ④. The computing device obtains a second evaluation result based on the evaluation results of a portion of the instructions under the second type of evaluation indicators and the evaluation results of another portion of the instructions under the second type of evaluation indicators.

[0165] In one possible scenario, the computing device selects, from the multiple instructions included in the instruction data, instructions whose scores under the first category of evaluation indicators are greater than or equal to a first score threshold, thereby obtaining a set of instructions to be evaluated. Based on the set of instructions to be evaluated and the evaluation rules, the computing device obtains a second evaluation result for the instructions. The first score threshold may be preset or user-defined based on actual application scenarios, and this application does not limit this.

[0166] FIG5 is a flow chart of another data evaluation method provided by the present application. As shown in FIG5 , a computing device first evaluates each instruction in the instruction data using each evaluation metric included in the first category of evaluation metrics to obtain a score for each instruction in the instruction data under each evaluation metric. Furthermore, the computing device filters out instructions with scores greater than or equal to a first score threshold from the instructions included in the instruction data, thereby obtaining a set of instructions to be evaluated. The computing device then uses the method described in S330 above to evaluate each instruction in the set of instructions to be evaluated using the second category of evaluation metrics to obtain a score for each instruction in the set of instructions to be evaluated under the second category of evaluation metrics. The computing device filters out instructions with scores greater than or equal to a second score threshold from the instructions included in the set of instructions to be evaluated. The second score threshold can be preset or user-defined based on actual application scenarios, and is not limited in this application. In this way, the number of instructions that the computing device needs to evaluate using the second category of evaluation metrics is reduced, the evaluation time for the instructions included in the instruction data is shortened, and the efficiency of evaluating the instructions included in the instruction data is improved.

[0167] In a possible scenario, the computing device displays at least one of the second evaluation result and a process of obtaining the second evaluation result.

[0168] The process of obtaining the second evaluation result refers to a process in which the computing device obtains the first evaluation result of the instruction under the second type of evaluation indicators.

[0169] Taking the evaluation indicator 1 in the second category of indicators as an example, the process of obtaining the second evaluation result of the instruction includes: instruction 1 is better than instruction a, instruction 1 is better than instruction b, instruction 1 is better than instruction c, and instruction 1 is worse than instruction d. The score of instruction c is score 1, and the score of instruction d is score 2. According to instructions c and d, the score of instruction 1 is determined to be score 3. Figure 6 is an example diagram of a display method of an evaluation result provided by the present application. As shown in Figure 6, the computing device simultaneously displays the second evaluation result and the process of obtaining the second evaluation result. In this way, it is convenient for the user to obtain the second evaluation result of the indicator obtained by the computing device under the second category of evaluation indicators, as well as the process of obtaining the second evaluation result, thereby improving the credibility of the second evaluation result of the instruction under the second category of evaluation indicators.

[0170] S340: The computing device obtains a score of the instruction according to the first evaluation result and the second evaluation result of the instruction.

[0171] For example, the computing device may use various methods to obtain the score of the instruction based on the first evaluation result and the second evaluation result of the instruction. In some possible examples, the score of the instruction may also be referred to as the evaluation result of the instruction.

[0172] For example, the computing device performs a sum operation on the score corresponding to the first evaluation result and the score corresponding to the second evaluation result, and directly uses the sum result as the score of the instruction.

[0173] In another example, the computing device performs a sum operation on the score corresponding to the first evaluation result and the score corresponding to the second evaluation result, multiplies, divides, adds, or subtracts a coefficient from the sum to obtain a calculation result, and the computing device uses the calculation result as the score of the instruction.

[0174] For example, the computing device performs a weighted operation on the score corresponding to the first evaluation result and the score corresponding to the second evaluation result to obtain the score of the instruction. Depending on the needs of the actual application, the computing device may also use other methods to obtain the score of the instruction based on the first evaluation result and the second evaluation result of the instruction, and this application is not limited to this.

[0175] In one possible scenario, after executing S340, the computing device may further execute S350. S350 specifically includes: the computing device selecting, from the multiple instructions, a target instruction associated with the target business based on the score of each instruction in the multiple instructions. The target instruction is used to indicate an instruction whose score under the second type of evaluation indicator is greater than or equal to a second score threshold, where the second score threshold is used to indicate a score threshold determined for the target business. The computing device also generates a model of the target business based on the target instruction.

[0176] In an embodiment of the present application, a computing device utilizes an AI model with semantic understanding capabilities to obtain, based on evaluation rules, evaluation criteria for instructions under the second category of evaluation indicators, and then evaluates instruction data based on the evaluation criteria under the second category of evaluation indicators. In this way, the computing device evaluates each instruction in the instruction data based on the evaluation criteria obtained by the model. The evaluation results for each instruction are generated under the same evaluation indicators, making the evaluation results for each instruction more comparable and improving the accuracy of the instruction evaluation.

[0177] The above describes the data evaluation method provided by this application in conjunction with the accompanying drawings. The following describes in detail the process of using the binary method to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators in conjunction with Figure 7, and describes in detail the process of using the sequential search method to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators in conjunction with Figure 8.

[0178] FIG7 is a flowchart of the first comparison method provided in the present application. As shown in FIG7 , the computing device may use the following process S71 to S75 to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators.

[0179] S71: The computing device selects a first reference example from a portion of instructions.

[0180] The score of the first reference example under the second type of evaluation index is the same as the median of the first score interval. The first score interval is used to indicate the interval formed by the scores of each instruction in a part of the instructions under the second type of evaluation index. Taking the scores of a part of the instructions under the second type of evaluation index shown in Table 2 above as an example, the first score interval is used to indicate the interval formed by score 1, score 2 to score j. If score 2 is the median of the first score interval, the computing device selects one of the texts corresponding to score 2 as the first reference example, such as the computing device selects text 2 as the first reference example.

[0181] In one possible scenario, the computing device may select multiple instructions to obtain multiple first reference examples. For example, the computing device selects text 2 and text 3 corresponding to score 2 as the first reference examples.

[0182] S72, the computing device compares the first reference sample with the instruction to be evaluated, and uses the AI ​​model to determine the pros and cons of the first reference sample and the instruction to be evaluated under the second type of evaluation indicators according to the evaluation rules.

[0183] When the computing device selects multiple instructions as multiple first reference examples, the computing device uses the AI ​​model to compare each of the multiple first reference examples with the instruction to be evaluated to determine how well the instruction to be evaluated compares to the multiple first reference examples under the second evaluation metric. This can reduce the impact of accidental factors on the quality of the instruction to be evaluated.

[0184] S73: If the first reference example is better than the instruction to be evaluated, the computing device selects a second reference example from a portion of the instructions.

[0185] The score of the second reference example under the second evaluation indicator is the same as the median of the second score range. The second score range is used to indicate the interval formed by the score of the instruction with the lowest score under the second evaluation indicator among a portion of instructions and the score of the first reference example under the second evaluation indicator.

[0186] S74: If the first reference example is inferior to the instruction to be evaluated, the computing device selects a third reference example from a portion of the instructions.

[0187] The score of the third reference example under the second type of evaluation indicator is the same as the median of the third score interval. The third score interval is used to indicate the interval formed by the score of the instruction with the highest score under the second type of evaluation indicator among a portion of instructions and the score of the first reference example under the second type of evaluation indicator.

[0188] S75, the computing device repeats S71 to S74 until the Mth reference sample and the Nth reference sample are obtained, and the score of the instruction to be evaluated under the second type of evaluation indicators is determined based on the Mth reference sample and the Nth reference sample.

[0189] The instruction to be evaluated is better than the Mth reference example, and no other instructions exist in a portion of the instructions, and the other instructions are better than the Mth reference example and worse than the Nth reference example. The instruction to be evaluated is worse than the Nth reference example, and no other instructions exist in a portion of the instructions, and the other instructions are worse than the Nth reference example and better than the Nth reference example. In this case, the computing device determines the score of the instruction to be evaluated under the second type of evaluation indicators based on the scores of the Mth reference example and the Nth reference example under the second type of evaluation indicators.

[0190] The computing device may determine the score of the instruction to be evaluated under the second type of evaluation indicators based on the scores of the Mth reference sample and the Nth reference sample under the second type of evaluation indicators in a variety of ways. Two possible examples are given below.

[0191] In Example 1, the computing device calculates the average of the scores of the Mth reference sample and the Nth reference sample under the second type of evaluation indicators, and uses the average as the score of the instruction to be evaluated under the second type of evaluation indicators.

[0192] In Example 2, a computing device performs a weighted operation on the scores of the Mth reference example and the Nth reference example under the second type of evaluation metric, and uses the result of the operation as the score of the instruction to be evaluated under the second type of evaluation metric. Depending on the needs of the actual application, the computing device may also use other methods to obtain the score of the instruction to be evaluated under the second type of evaluation metric based on the scores of the Mth reference example and the Nth reference example under the second type of evaluation metric.

[0193] In some possible scenarios, the computing device repeats steps S71 to S74 above, completes the comparison of the instruction to be evaluated with each instruction in the portion of instructions, and is unable to obtain the Mth reference instruction and the Nth reference instruction from the portion of instructions. Depending on the relationship between the expected performance of the instruction to be evaluated under the second type of evaluation metric and the scores of the portion of instructions under the second type of evaluation metric, the computing device may use different methods to obtain the scores of the instruction to be evaluated under the second type of evaluation metric, which are described below for different scenarios.

[0194] In case A, the computing device determines that the expected performance of the instruction to be evaluated under the second type of evaluation index is worse than the instruction with the smallest score under the second type of evaluation index included in a portion of instructions. Depending on the requirements of the actual application, the computing device can perform the following (1) or (2).

[0195] (1) The computing device determines a second evaluation result of the instruction to be evaluated under the second type of evaluation indicator based on the instruction with the smallest score under the second type of evaluation indicator included in a portion of instructions.

[0196] (2) The computing device displays a second prompt message and receives a second evaluation result of the instruction to be evaluated by the user under the second type of evaluation indicator. The second prompt message is used to prompt the user to evaluate the instruction to be evaluated under the second type of evaluation indicator.

[0197] In case B, the computing device determines that the expected performance of the instruction to be evaluated under the second type of evaluation index is better than the instruction with the highest score under the second type of evaluation index included in a portion of instructions. Depending on the needs of the actual application, the computing device can perform the following (1) or (2).

[0198] (1) The computing device determines a second evaluation result of the instruction to be evaluated under the second evaluation metric based on the instruction with the highest score under the second evaluation metric included in a portion of the instructions. Regarding (2), please refer to the relevant content in Scenario A above and will not be repeated here.

[0199] The above describes the process of using the binary method to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators. The following describes in detail the process of using the sequential search method to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators in conjunction with Figure 8.

[0200] FIG8 is a flowchart of the second comparison method provided in the present application. As shown in FIG8 , the computing device may use the following process S81 to S85 to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators.

[0201] S81, the computing device selects a first reference sample from a portion of instructions.

[0202] The score of the first reference example under the second type of evaluation indicator is the same as the minimum score of the first score interval. For the relevant content of the first score interval, please refer to the relevant description in Figure 7 above, which will not be repeated here. If the scores 1 to j in the first score interval gradually increase, in this case, the computing device selects any text with a score of 2 as the first reference example, such as selecting text 2 as the first reference example.

[0203] In a possible scenario, the computing device may select multiple instructions as multiple first reference examples, such as selecting text 2 and text 3 as first reference examples.

[0204] S82, the computing device uses the AI ​​model to compare the first reference sample with the instruction to be evaluated, and determines the pros and cons of the first reference sample and the instruction to be evaluated under the second type of evaluation indicators according to the evaluation rules.

[0205] When the computing device selects multiple instructions as multiple first reference examples, the computing device uses the AI ​​model to compare each of the multiple first reference examples with the instruction to be evaluated to determine the relative merits of the instruction to be evaluated and the multiple first reference examples under the second evaluation metric. This can reduce the impact of accidental factors on the relative merits of the instruction to be evaluated.

[0206] S83, if the first reference example is better than the instruction to be evaluated, the computing device may execute the following (1) or (2) to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators.

[0207] (1) The computing device determines the score of the first reference example under the second type of evaluation indicator as the score of the instruction to be evaluated under the second type of evaluation indicator. Regarding (2), please refer to the relevant content of Scenario A in FIG7 above and will not be repeated here.

[0208] S84: If the first reference example is inferior to the instruction to be evaluated, the computing device selects a second reference example from a portion of the instructions.

[0209] The score of the second reference example under the second type of evaluation indicators is the same as the minimum score of the second score interval. The second score interval is used to indicate that other instructions include the interval consisting of the score of the first instruction and the score of the second instruction under the second type of evaluation indicators. Other instructions refer to instructions other than the instructions corresponding to the first reference example included in a part of instructions. The first instruction refers to the instruction with the lowest score under the second type of evaluation indicators among the other instructions, and the second instruction refers to the instruction with the highest score under the second type of evaluation indicators among the other instructions.

[0210] S85, the computing device repeats S81 to S84 until the Mth reference sample and the Nth reference sample are determined, and the score of the instruction to be evaluated under the second type of evaluation indicators is determined based on the Mth reference sample and the Nth reference sample.

[0211] For details about this process, please refer to the relevant description in FIG7 above, which will not be repeated here.

[0212] In one possible scenario, the computing device repeats S81 to S84 and has completed the comparison of the instruction to be evaluated with each instruction in the portion of instructions. Furthermore, the computing device determines that the expected performance of the instruction to be evaluated under the second evaluation metric is superior to the instruction in the portion of instructions that has the highest score under the second evaluation metric. In this scenario, the computing device can use the method described in Scenario B in FIG7 to determine the score of the instruction to be evaluated under the second evaluation metric. For details, please refer to the above description and will not be repeated here.

[0213] Figure 9 is a flow chart of the data evaluation method provided by the present application. As shown in Figure 9, the computing device evaluates the multiple instructions included in the instruction data under the first type of evaluation indicators to obtain a first evaluation result of each instruction in the multiple instructions under the first type of evaluation indicators. And the computing device receives the user's evaluation results of a part of the instructions in the instruction data to obtain a data sample. The computing device selects some instructions (such as instruction x1, instruction y1) from the data sample, and constructs a comparison sample according to the evaluation rules. The computing device selects a reference sample from the data sample, and based on the reference sample and the comparison sample, performs data evaluation on another part of the instructions in the instruction data under the second type of evaluation indicators to obtain a second evaluation result of the other part under the second type of evaluation indicators. The computing device comprehensively evaluates based on the first evaluation result and the second evaluation result to obtain the evaluation result of the instruction.

[0214] The computing device can execute the data evaluation method provided in this application using the sequence shown in FIG10 . FIG10 is a schematic diagram of an execution timing provided in this application. As shown in FIG10 , the computing device receives the user's evaluation results of a portion of instructions in the instruction data under the second type of evaluation indicators, and obtains a data sample based on the evaluation results of the portion of instructions and the portion of instructions under the second type of evaluation indicators. The computing device selects instructions from the data sample to construct a comparison sample, and the computing device inputs the comparison sample into the model. The computing device uses the model to evaluate another portion of the instructions in the instruction data. If the score of the instruction to be evaluated is within the score range, the computing device obtains the evaluation result of the instruction to be evaluated under the second type of evaluation indicators based on the evaluation results of the portion of instructions under the second type of evaluation indicators. The instruction to be evaluated is one of the instructions in the other portion. If the score of the instruction to be evaluated is outside the score range, the computing device prompts the user to evaluate the instruction to be evaluated under the second type of evaluation indicators.

[0215] The data evaluation method provided by this application is described above in conjunction with the accompanying drawings. The data evaluation device provided by this application is described below in conjunction with the accompanying drawings. Figure 11 is a schematic structural diagram of the data evaluation device provided by this application. As shown in Figure 11, data evaluation device 1100 includes: a transceiver module 1110, a first evaluation module 1120, a second evaluation module 1130, and a processing module 1140.

[0216] The transceiver module 1110 is used to obtain instruction data. The instruction data includes multiple instructions. The first evaluation module 1120 is used to obtain a first evaluation result for each of the multiple instructions based on the instruction data and the first type of evaluation indicators. The first evaluation result of the instruction includes the score of the instruction under the first type of evaluation indicators. The first type of evaluation indicators are used to describe the attributes of the instruction. The second evaluation module 1130 is used to obtain a second evaluation result for each of the multiple instructions based on the instruction data and the evaluation rules using the AI ​​model. The second evaluation result of the instruction includes the score of the instruction under the second type of evaluation indicators. The second type of evaluation indicators are used to describe the characteristics of the instruction. The evaluation rules are used to indicate the degree of pros and cons of different instructions under the second type of evaluation indicators. The processing module 1140 is used to obtain the score of the instruction based on the first evaluation result and the second evaluation result of the instruction.

[0217] In one possible scenario, the data evaluation device 1100 further includes a display module 1150. The display module 1150 is configured to display a first prompt message. The first prompt message is configured to prompt a user to evaluate a portion of the multiple instructions under a second type of evaluation indicator. The transceiver module 1110 is further configured to receive the user's evaluation results of the portion of instructions under the second type of evaluation indicator based on the first prompt message. The second evaluation module 1130 is specifically configured to obtain, based on the evaluation rules and the evaluation results of the portion of instructions under the second type of evaluation indicator, the evaluation results of another portion of the multiple instructions under the second type of evaluation indicator. The second evaluation module 1130 is further specifically configured to obtain a second evaluation result based on the evaluation results of the portion of instructions under the second type of evaluation indicator and the evaluation results of the other portion of instructions under the second type of evaluation indicator.

[0218] In one possible scenario, second evaluation module 1130 is specifically configured to construct a comparison example using at least two instructions from a portion of instructions according to evaluation rules. The comparison example indicates the degree of superiority or inferiority of the at least two instructions under a second type of evaluation metric. Second evaluation module 1130 is further specifically configured to obtain evaluation results for another portion of the plurality of instructions under the second type of evaluation metric based on the comparison example and the evaluation results for the portion of instructions under the second type of evaluation metric.

[0219] In one possible scenario, the quality of at least two instructions indicated by the comparison sample under the second type of evaluation indicators reflects the standard for judging the quality of the instructions. The second evaluation module 1130 is specifically used to compare the quality of the instruction to be evaluated with the quality of each instruction in a part of the instructions according to the standard for judging the quality of the instructions reflected by the comparison sample. The instruction to be evaluated is used to indicate an instruction in another part of the instructions. Depending on the difference in the quality of the expected performance of the instruction to be evaluated under the second type of evaluation indicators and the quality of the evaluation results of each instruction in a part of the instructions under the second type of evaluation indicators, the data evaluation device 1100 can have different processing methods, which are explained in different situations below.

[0220] Case 1: The expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the worst evaluation result among a part of instructions, and the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the best evaluation result among a part of instructions.

[0221] In this case, the second evaluation module 1130 is further specifically configured to determine the evaluation result of the instruction to be evaluated under the second type of evaluation indicators according to the evaluation results of a portion of instructions under the second type of evaluation indicators.

[0222] Case 2: The expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the best evaluation result among a part of instructions, or the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the worst evaluation result among a part of instructions.

[0223] In this case, the display module 1150 is further configured to display a second prompt message. The second prompt message is configured to prompt the user to evaluate the instruction to be evaluated under the second type of evaluation indicators.

[0224] In case 3, the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the best evaluation result among a part of the instructions.

[0225] In this case, the second evaluation module 1130 is specifically configured to determine the evaluation result of the instruction to be evaluated according to the evaluation result of the instruction with the best evaluation result among a portion of instructions.

[0226] Case 4: The expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the worst evaluation result among a part of the instructions.

[0227] In this case, the second evaluation module 1130 is specifically configured to determine the evaluation result of the instruction to be evaluated according to the evaluation result of the instruction with the worst evaluation result among a portion of instructions.

[0228] In one possible scenario, the second evaluation module 1130 is specifically used to: select instructions whose scores under the first type of evaluation indicators are greater than or equal to the first score threshold from the multiple instructions included in the instruction data, obtain a set of instructions to be evaluated, and obtain a second evaluation result for each instruction in the set of instructions to be evaluated based on the evaluation rules and the second type of evaluation indicators.

[0229] In one possible scenario, processing module 1140 is further configured to select, from the plurality of instructions, a target instruction associated with the target business based on the score of each instruction in the plurality of instructions. The target instruction indicates an instruction whose score under the second type of evaluation metric is greater than or equal to a second score threshold, where the second score threshold indicates a score threshold determined for the target business. Processing module 1140 is further configured to generate a model of the target business based on the target instruction.

[0230] In a possible scenario, the display module 1150 is further configured to display at least one of the first evaluation result, the process of obtaining the first evaluation result, the second evaluation result, and the process of obtaining the second evaluation result.

[0231] In a possible scenario, the display module 1150 is further configured to display third prompt information. The third prompt information is configured to prompt the user to input at least one of the following: instruction data, evaluation rules, first-type evaluation indicators, and second-type evaluation indicators.

[0232] It is worth noting that if the data evaluation device is implemented through a software module, for example, the software module can be provided to users through a cloud service subscription model, and users can choose different subscription levels according to their needs; for example, the software module can also provide enterprise-level customization services such as professional domain customization, interface personalization and extended functions according to the needs of users or enterprises.

[0233] In addition, the data evaluation device provided in this application can also be provided to users as a value-added service, which is not limited in this application. When the data evaluation device is implemented as a software module, the data evaluation device can also be embedded in a tool chain system of a model with semantic understanding capabilities, which can be a large model GPT-4, other large language models (large models), other neural network models, etc.

[0234] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and storage medium can also exist as discrete components in a network device or a terminal device.

[0235] The present application also provides a chip. The chip includes an interface circuit and a control circuit. The interface circuit is used to obtain instruction data and cooperate with the control circuit to execute the data evaluation method.

[0236] Embodiments of the present application also provide a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform a data evaluation method.

[0237] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data evaluation method.

[0238] The present application also provides a chip system including a processor for implementing the functions of the computing device in the above method. In one possible design, the chip system also includes a memory for storing program instructions and / or data. The chip system can be composed of a chip alone or include a chip and other discrete devices.

[0239] In the above embodiments, they can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computing device, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computing device can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0240] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data evaluation method, characterized in that: The method is performed by a computing device, on which an AI model is deployed, and the AI ​​model has semantic understanding capabilities. The method includes: Acquire instruction data; the instruction data includes multiple instructions; Obtaining a first evaluation result for each of the plurality of instructions; wherein the first evaluation result of the instruction includes a score of the instruction under a first type of evaluation indicator, where the first type of evaluation indicator is used to describe an attribute of the instruction; Obtaining, using the AI ​​model, a second evaluation result for each of the plurality of instructions based on the instruction data and the evaluation rule; wherein the second evaluation result for the instruction includes a score for the instruction under a second type of evaluation indicator, the second type of evaluation indicator being used to describe characteristics of the instruction, and the evaluation rule being used to indicate the degree of superiority or inferiority of different instructions under the second type of evaluation indicator; A score of the instruction is obtained according to the first evaluation result and the second evaluation result of the instruction.

2. The method according to claim 1, characterized in that The obtaining, using the AI ​​model and according to the instruction data and the evaluation rule, a second evaluation result of each of the plurality of instructions includes: displaying first prompt information, where the first prompt information is used to: prompt the user to evaluate a portion of the plurality of instructions under the second type of evaluation indicators; receiving an evaluation result of the user on the portion of instructions based on the first prompt information and under the second type of evaluation indicators; Obtaining, using the AI ​​model, evaluation results of another portion of the plurality of instructions under the second type of evaluation indicators based on the evaluation rules and the evaluation results of the portion of instructions under the second type of evaluation indicators; The second evaluation result is obtained according to the evaluation results of the part of instructions under the second type of evaluation indicators and the evaluation results of the other part of instructions under the second type of evaluation indicators.

3. The method according to claim 2, characterized in that The obtaining, using the AI ​​model, evaluation results of another portion of the plurality of instructions under the second type of evaluation indicators according to the evaluation rules and the evaluation results of the portion of instructions under the second type of evaluation indicators, includes: Using at least two instructions from the portion of instructions, constructing a comparison example according to the evaluation rule; the comparison example is used to indicate the degree of superiority or inferiority of the at least two instructions under the second type of evaluation indicators; Utilizing the AI ​​model, based on the comparison examples and the evaluation results of the part of instructions under the second type of evaluation indicators, obtain the evaluation results of another part of the multiple instructions under the second type of evaluation indicators.

4. The method according to claim 3, characterized in that Obtaining evaluation results of another portion of the plurality of instructions under the second type of evaluation indicators includes: If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the worst evaluation result among the part of instructions, and the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the best evaluation result among the part of instructions, the evaluation result of the instruction to be evaluated under the second type of evaluation indicators is determined according to the evaluation results of the part of instructions under the second type of evaluation indicators; the instruction to be evaluated is one of the other part of instructions.

5. The method according to claim 4, characterized in that The method further comprises: If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the best evaluation result among the part of instructions, or the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the worst evaluation result among the part of instructions, a second prompt message is displayed; the second prompt message is used to: prompt the user to evaluate the instruction to be evaluated under the second type of evaluation indicators.

6. The method according to claim 4, characterized in that The method further comprises: If the expected performance of the instruction to be evaluated under the second type of evaluation indicators is better than the instruction with the best evaluation result among the part of instructions, determining the evaluation result of the instruction to be evaluated according to the evaluation result of the instruction with the best evaluation result among the part of instructions; Alternatively, if the expected performance of the instruction to be evaluated under the second type of evaluation indicators is worse than the instruction with the worst evaluation result among the part of instructions, the evaluation result of the instruction to be evaluated is determined according to the evaluation result of the instruction with the worst evaluation result among the part of instructions.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining, according to the instruction data and the evaluation rule, a second evaluation result of each of the plurality of instructions comprises: Selecting, from the plurality of instructions included in the instruction data, instructions whose scores under the first type of evaluation indicators are greater than or equal to a first score threshold, to obtain a set of instructions to be evaluated; The second evaluation result is obtained according to the instruction set to be evaluated and the evaluation rule.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Selecting a target instruction associated with a target business from the multiple instructions based on the score of each instruction in the multiple instructions; the target instruction is used to indicate an instruction whose score under the second type of evaluation indicator is greater than or equal to a second score threshold, and the second score threshold is used to indicate a score threshold determined for the target business; A model of the target business is generated according to the target instruction.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: At least one of the first evaluation result, a process of obtaining the first evaluation result, the second evaluation result, and a process of obtaining the second evaluation result is displayed.

10. The method according to any one of claims 1 to 9, characterized in that Before obtaining the instruction data, the method further includes: Display a third prompt message; the third prompt message is used to prompt the user to input at least one of the instruction data, the evaluation rules, the first type of evaluation indicators, and the second type of evaluation indicators.

11. A data evaluation device, characterized in that: The data evaluation device comprises: The transceiver module is used to: obtain instruction data; the instruction data includes multiple instructions; A first evaluation module is configured to obtain a first evaluation result of each of the plurality of instructions, wherein the first evaluation result of the instruction includes a score of the instruction under a first type of evaluation indicator, where the first type of evaluation indicator is used to describe an attribute of the instruction; a second evaluation module, configured to: utilize the AI ​​model to obtain, based on the instruction data and the evaluation rules, a second evaluation result for each of the plurality of instructions; wherein the second evaluation result of the instruction includes a score of the instruction under a second type of evaluation indicator, the second type of evaluation indicator being used to describe characteristics of the instruction, and the evaluation rules being used to indicate the degree of superiority or inferiority of different instructions under the second type of evaluation indicator; The processing module is used to obtain a score of the instruction according to the first evaluation result and the second evaluation result of the instruction.

12. A chip, characterized in that: It comprises an interface circuit and a control circuit; the interface circuit is used to obtain instruction data and cooperate with the control circuit to execute the method according to any one of claims 1 to 10.

13. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores a program code, and when the processor executes the program code, the processor is configured to execute the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that The computer readable storage medium stores computer program instructions; When the computer program instructions are executed in a computing device, the computing device is caused to perform the method according to any one of claims 1 to 10.

15. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 10.

Images