Information processing link evaluation method and apparatus, and device, medium and program product

Abstract

According to the embodiments of the present disclosure, provided are an information processing link evaluation method and apparatus, and a device, a medium and a program product. The method comprises: receiving a data query request with respect to a first processing node among one or more processing nodes comprised in an information processing link, wherein the one or more processing nodes are each configured to use a machine learning model to execute information processing; on the basis of a query result corresponding to the data query request, generating a data resource with respect to the first processing node, wherein the data resource comprises one or more data entries, and each data entry comprises an input of the first processing node; receiving model configuration information with respect to at least one processing node to be evaluated among the one or more processing nodes, wherein the at least one processing node at least comprises the first processing node; and generating a data object on the basis of the data resource and the model configuration information, wherein the data object is used for constructing a first evaluation case with respect to the at least one processing node. Therefore, case evaluation for a series link can be quickly and conveniently implemented.

Description

Information processing link evaluation method, device, equipment, medium and program product Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to information processing link evaluation methods, apparatuses, devices, computer-readable storage media, and computer program products. Background Art

[0002] In today's technological development, with the continuous advancement of artificial intelligence, the application of technologies such as machine learning models is becoming increasingly widespread. In this era of machine learning model application, prompt engineering (PE) has a rapid iteration speed. When testing prompts, it is necessary to cover a wider range of data samples to ensure higher credibility of the conclusions drawn.

[0003] Summary of the Invention

[0004] In a first aspect of the present disclosure, a method for evaluating an information processing link is provided. The method comprises: receiving a data query request for a first processing node among one or more processing nodes included in an information processing link, wherein the one or more processing nodes are each configured to perform information processing using a machine learning model; generating a data resource for the first processing node based on a query result corresponding to the data query request, wherein the data resource comprises one or more data entries, each data entry comprising an input of the first processing node; receiving model configuration information for at least one processing node to be evaluated among the one or more processing nodes, wherein the at least one processing node comprises at least the first processing node; and generating a data object based on the data resource and the model configuration information, wherein the data object is used to construct a first evaluation case for the at least one processing node.

[0005] In a second aspect of the present disclosure, an information processing link evaluation device is provided. The device includes: a query request receiving module configured to receive a data query request for a first processing node among one or more processing nodes included in the information processing link, wherein the one or more processing nodes are respectively configured to perform information processing using a machine learning model; a data resource generating module configured to generate a data resource for the first processing node based on a query result corresponding to the data query request, wherein the data resource includes one or more data entries, each data entry including an input of the first processing node; a configuration information receiving module configured to receive model configuration information for at least one processing node to be evaluated among the one or more processing nodes, wherein the at least one processing node includes at least the first processing node; and a data object generating module configured to generate a data object based on the data resource and the model configuration information, wherein the data object is used to construct a first evaluation case for the at least one processing node.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] 2A to 2H are schematic diagrams showing example interfaces of information processing link evaluation according to some embodiments of the present disclosure;

[0013] 3A to 3C are schematic diagrams illustrating example interfaces for executing use cases according to some embodiments of the present disclosure;

[0014] FIG4 shows a flowchart of an example process of information processing link evaluation according to some embodiments of the present disclosure;

[0015] FIG5 shows a schematic structural block diagram of an example apparatus for information processing link evaluation according to certain embodiments of the present disclosure; and

[0016] FIG6 illustrates a block diagram showing an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0017] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0018] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0019] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0021] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0022] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0024] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0026] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", and these terms are used interchangeably in this article. A model can also include different types of processing units or networks.

[0027] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also known as the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also known as input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0028] As used herein, the term "component" may refer to any suitable model, module, unit, etc. used to implement a special effect. Such a component may provide a corresponding output based on the input provided and may include any suitable operation, calculation, etc. An example of a component is an algorithm. Below, some embodiments of the present disclosure will be primarily described with reference to algorithms, but it should be understood that such embodiments are also applicable to other types of components.

[0029] As briefly mentioned above, in the era of machine learning models (also known as large models), prompt word engineering iterations are rapid. However, when manually testing prompt words, the sample coverage is insufficient, resulting in poorly credible conclusions drawn from individual examples. To address this issue, automated evaluation platforms have been introduced for automated evaluation, but this approach still presents some challenges.

[0030] One issue is the high cost of creating and maintaining datasets. For example, when constructing time-sensitive examples, it's necessary to pull the latest data in real time and accumulate it into a new dataset. The high cost of bad examples in certain links requires obtaining them from news feedback documents.

[0031] Another issue is the high cost of creating and maintaining evaluation use cases. When the PE versions, datasets, and evaluation rules used in a use case are inconsistent, multiple use cases must be created and individually bound to evaluation sets and rules. Furthermore, use cases do not support rapid duplication or modification of bound machine learning models.

[0032] Specifically, when constructing a data set, it is necessary to manually construct multiple versions of the data set. For example, the first version constructs a data set for normal cases, and then a data set for bad cases, and then data sets of different numbers (such as 100, 50 or other numbers) may be constructed, and so on. The second step may be to construct rules, including rules for automatic scoring of machine learning models, and / or manual scoring rules. After the first two steps are completed, the third step is to create use cases. Here, the use case needs to bind the data set and rules, but different use cases may do different things. For example, when calling PE version 1, the first data set plus automatic scoring rules and manual working rules are used to run the data; when calling PE version 2, the second data set plus automatic scoring rules and manual working rules are used to run the data, and so on.

[0033] Another issue is the high cost of manual evaluation. The evaluation results of multiple use cases need to be manually compiled into an evaluation document for subsequent scoring. Multiple people then score the multiple use case evaluation results based on the evaluation document. Finally, the manual and machine scores are aggregated to calculate key metrics (such as full score rate).

[0034] Another issue is the lack of full-link / multi-link evaluation capabilities. For example, in the news and information sector, the current model only supports evaluation of search, summarization, and reporting. However, there is still a need for full-link evaluation to verify the effectiveness of the entire chain, including search, summarization, and reporting.

[0035] In view of this, an embodiment of the present disclosure proposes an improved scheme for information processing link evaluation. According to various embodiments of the present disclosure, a data query request for a first processing node among one or more processing nodes included in an information processing link is received, and the one or more processing nodes are respectively configured to perform information processing using a machine learning model. Based on the query result corresponding to the data query request, a data resource for the first processing node is generated, the data resource including one or more data entries, each data entry including an input of the first processing node. Model configuration information for at least one processing node to be evaluated among the one or more processing nodes is received, and the at least one processing node includes at least the first processing node. A data object is generated based on the data resource and the model configuration information, and the data object is used to construct a first evaluation case for at least one processing node.

[0036] Thus, using the data from the first processing node and based on the model configuration information of at least one processing node, a use case evaluation of the series link can be achieved. This approach allows for quick and convenient establishment of data resources, flexible completion of the overall evaluation of the series link, and effective reduction in evaluation costs.

[0037] Sample Environment

[0038] FIG1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Example environment 100 includes a use case management platform 110, with which users 140 can interact. Use case management platform 110 is configured to perform information processing link assessments. An information processing link can refer to a complete path in an information processing process, through a single link or a series of sequential links and steps, that ultimately achieves a specific information processing goal.

[0039] In some embodiments, the information processing link may include one or more processing nodes connected in series for information processing (for example, processing node 121-1, processing node 121-2, processing node 121-3, ..., processing node 121-N, where N is an integer greater than or equal to 1). It should be understood that information processing links for different scenarios may have different types and numbers of processing nodes. Taking the news information evaluation scenario as an example, the corresponding information processing link may include, for example, a search node, a summary node, a broadcast node, etc.

[0040] In some embodiments, each processing node may be configured with one or more corresponding machine learning models (in the figure, each processing node is configured with a corresponding machine learning model), and the machine learning model may be configured to perform information processing on the corresponding processing node. For example, processing node 121-1, processing node 121-2, processing node 121-3, ..., processing node 121-N may be respectively configured with machine learning model 122-1, machine learning model 122-2, machine learning model 122-3, ..., machine learning model 122-N. The type of machine learning model can be set according to the actual usage scenario.

[0041] The use case management platform 110 can communicate with the use case execution platform 130. The use case execution platform 130 can be configured to provide services related to use cases based on data from the use case management platform 110. Users 140 can interact with the use case execution platform 130.

[0042] In some embodiments, the use case management platform 110 and the use case execution platform 130 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.). The server 130 can be various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0043] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0044] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0045] Example Interaction

[0046] An example interaction process according to an embodiment of the present disclosure will be described below with reference to the accompanying drawings.

[0047] Figures 2A through 2H illustrate example interfaces 200A through 200H for information processing link evaluation according to some embodiments of the present disclosure. For ease of discussion, these embodiments will be described in conjunction with environment 100 of Figure 1 . In embodiments of the present disclosure, interfaces 200A through 200H may be provided by use case management platform 110 in environment 100 of Figure 1 .

[0048] As shown in Figures 2A-2C, the use case management platform 110 receives a data query request for the first processing node 120-n (n ranges from 1 to N, and n is an integer) in one or more processing nodes included in the information processing link (for example, it may include processing node 121-1, processing node 121-2, ..., processing node 121-N, where N is an integer greater than or equal to 1). Exemplarily, an information processing link can refer to a complete path in the information processing process that ultimately achieves a specific information processing goal through a single link or a series of sequential links and steps. This link may include one or more processing nodes connected in series for information processing. Processing nodes may, for example, be collection nodes, transmission nodes, storage nodes, analysis nodes, integration nodes, conversion nodes, etc. It should be understood that information processing links for different scenarios may have different types and numbers of processing nodes. Taking the news information evaluation scenario as an example, the information processing link may include a news information evaluation link. The processing nodes of the news information evaluation link may, for example, include a continuous search node, a summary node, a broadcast node, etc. For ease of understanding, the following will continue to discuss the embodiments of the present disclosure using the news information evaluation scenario as an example.

[0049] Here, one or more processing nodes are respectively configured to perform information processing using a machine learning model 122 (which may include one or more machine learning models, for example, machine learning model 122-1, machine learning model 122-2, ..., machine learning model 122-N, where N is an integer greater than or equal to 1). It should be understood that each processing node can have several machine learning models for performing the information processing functions required to be implemented by the corresponding processing node. In the example of a news information evaluation scenario, assuming that some news is to be searched for user 140, the corresponding machine learning model can be used at the search node of the news information evaluation link to search for a certain number of news that meet the requirements from the collected news information. If the searched news is to be summarized, the corresponding machine learning model can be used to summarize the news at the summary node. It should be understood that one or more news information can be summarized into a summary news. If the summarized one or more summary news are to be broadcast, the corresponding machine learning model can be used at the broadcast node to implement the news broadcast.

[0050] In some embodiments, the data query request may indicate query conditions for samples processed by the first processing node 120-n. The first processing node may correspond to a certain number of samples (also referred to as cases) to be processed. In some examples, the content of the samples may be selected based on actual needs. In the example of a news information evaluation scenario, the samples to be processed may include, for example, multiple timely news information, multiple economic news information, multiple educational news information, and the like.

[0051] 2A , illustratively, the use case management platform 110 may receive query conditions via a query interface 200A. For example, if the first processing node is a summary node, the user 140 may select a module labeled "Summary Content Query" 210 on the use case management platform 110 to set query conditions on the corresponding page. In some embodiments, the query conditions may include one or more constraints for the sample to be queried. In some embodiments, to receive the query conditions, the use case management platform 110 may present corresponding setting entries for one or more constraints (e.g., entry 211, entry 212, entry 213, etc.), and receive settings for the one or more constraints via the corresponding setting entries for the one or more constraints.

[0052] In some embodiments, one or more constraints may include keywords included in the sample to be queried. As an example, entry 211 may receive settings for keywords included in the sample to be queried. For example, assuming that the sample to be queried includes multiple news information on education, the keywords may be set to "XX University", "XX Examination" or other keywords, etc. Exemplarily, in entry 211, the setting of keywords may be by selecting keywords or directly entering keywords, etc., and there is no limitation here.

[0053] Alternatively or additionally, in some embodiments, one or more constraints may include a time range corresponding to the sample to be queried. As an example, entry 212 may receive a setting for the time range corresponding to the query. The time range may specifically include a date range, a time period range, etc. For example, assuming that the sample to be queried includes multiple news information with timeliness, the time range may be set to, for example, "X year X month X day - Y year Y month Y day", "X month X day X hour - X month X day Y hour", or other time ranges.

[0054] Alternatively or additionally, in some embodiments, one or more constraints may include the number of samples to be queried. As an example, the portal 213 may receive a setting for the number of samples to be queried. For example, assuming that the samples to be queried include multiple news information on the economy, the number of samples to be queried may be set to 50, 100, or another number through the portal 213. In some examples, the number that can be set may not exceed the total number of samples to be queried. In other examples, if the number set by the user 140 through the portal 213 exceeds the total number of samples to be queried, the use case management platform 110 may display all samples to be queried in the total number and prompt the user 140 accordingly.

[0055] It should be understood that the one or more constraints for the query examples may include other constraints in addition to the constraints discussed above. Furthermore, the above embodiments of the one or more constraints are merely examples and are not intended to be limiting. The one or more constraints for the query examples may be set based on specific information processing scenarios.

[0056] Continuing with FIG. 2A , in some embodiments, based on the query conditions, the use case management platform 110 may generate query instructions 215 corresponding to the query conditions. The specific instruction content shown in the figure is merely an example, provided for ease of understanding, and is not intended to be limiting. For example, the query instructions 215 may be program statements, such as SQL (Structured Query Language) statements or other program statements, without limitation.

[0057] In traditional solutions, users are required to manually enter program statements, etc., into the use case management platform to search for examples. However, according to embodiments of the present disclosure, referring to Figures 2A and 2B , based on a query instruction 215 generated and presented on query interface 200A, use case management platform 110 can receive a copy operation of query instruction 215 from user 140 on query interface 200A, and then automatically paste query instruction 215 into a predetermined location on instruction page 200B for execution. Thus, referring to Figure 2C , in some embodiments, use case management platform 110 can obtain query results 220 by executing query instruction 215.

[0058] By setting query conditions for the samples to be queried in the above embodiment, the problems of poor data reusability and high data preparation costs can be solved, and data resources can be quickly obtained. In addition, by automatically generating query instructions based on the query conditions, the query rate can be greatly improved and the threshold for using the use case management platform can be lowered. In addition, the efficiency and accuracy of the information processing link can play a key role in realizing the value of information.

[0059] Furthermore, based on the query result 220 corresponding to the data query request, the use case management platform 110 generates a data resource for the first processing node 120-n. The data resource includes one or more data entries (for example, data entry 221-1, data entry 221-2, data entry 221-3, etc., which may also be referred to individually or collectively as data entries 221). Each data entry 221 includes the input of the first processing node 120-n. The data resource may also be included in a data set. For different query results generated according to different constraints, different data resources may be generated, and accordingly, different data sets may also be generated. In some embodiments, the input of the first processing node 120-n may include samples queried according to one or more constraints based on the query conditions. As an example, one data entry 221 may correspond to one queried sample.

[0060] Furthermore, the use case management platform 110 receives model configuration information for at least one processing node to be evaluated from one or more processing nodes, the at least one processing node including at least a first processing node 120-n. As discussed above, one or more processing nodes in the information processing chain may be connected in series in a sequential manner. In some embodiments, the at least one processing node may include a plurality of processing nodes arranged in a sequential manner, and the first processing node 120-n may be a processing node that is ranked first among the plurality of processing nodes. For example, if the at least one processing node includes a summary node and a broadcast node, the summary node is the first processing node 120-n. It should be understood that if the at least one processing node includes only one processing node, the processing node may be the first processing node 120-n.

[0061] In some embodiments, the model configuration information of the target processing node in at least one processing node may include identification information of the target machine learning model for the target processing node. Exemplarily, each processing node in at least one processing node (such as the target processing node) may correspond to one or more machine learning models (such as the target machine learning model). In the following, the processing node and / or the machine learning model corresponding to the processing node may also be referred to as a robot (Bot). Each processing node may have its own model configuration information. Each machine learning model may have corresponding identification information, such as an identifier (ID) or other identification information.

[0062] 2D , taking the example that at least one processing node to be evaluated includes a summary node and a broadcast node, the use case management platform 110 can present a link node configuration interface 200D for "summary and broadcast evaluation". On the link node configuration interface 200D, the use case management platform 110 can receive the user 140's configuration of the data set ID 231 (i.e., the identifier of the data resource), the summary bot ID 232 (i.e., the identifier of the machine learning model corresponding to the summary node), and the broadcast bot ID 233 (i.e., the identifier of the machine learning model corresponding to the broadcast node). Exemplarily, the configuration method may include a selection method, an input method, or other methods. For example, the use case management platform 110 can receive the user 140's selection of the data set ID, or the input of the summary bot ID 232 and the broadcast bot ID 233, etc., without limitation here. In some embodiments, if the user 140 has not configured the model configuration information of a processing node in at least one processing node, the Bot ID of the online link (i.e., the identifier of the machine learning model) can be used for the processing node.

[0063] Furthermore, the use case management platform 110 generates a data object based on the data resources and model configuration information, and the data object is used to construct a first evaluation case for at least one processing node. Exemplarily, the data object may also be a data set file, in which the data set file may be a file in the comma-separated value (CSV) format. CSV files are generally plain text files. Referring to FIG2E , the data object is represented by a CSV file. After the use case management platform 110 generates the CSV file based on the data resources and model configuration information, a prompt window 240 may be presented on the link node configuration interface 200D to remind the user 140 that the CSV file has been successfully created. It should be understood that the user 140 may also be reminded in any other appropriate manner that the data object has been generated.

[0064] In some embodiments, when downloading a data object, the format of the data object can be converted to the file format and content format required by an evaluation platform such as the use case management platform 110. Referring to FIG2C , a data object can be downloaded using a download button 245. This improves the efficiency of data object format conversion. A unified data format can facilitate the subsequent evaluation process.

[0065] In some embodiments, to generate a data object based on a data resource and model configuration information, the use case management platform 110 may add identification information to the data resource as a target field value of a target field in the data resource, and may generate a data object based on the data resource to which the identification information is added. In such an embodiment, the data resource may include multiple fields, and the field corresponding to the model configuration information may be a target field among the multiple fields. For example, still taking the example of at least one processing node to be evaluated including a summary node and a broadcast node, if the identification information is an identifier, the use case management platform 110 may write the summary bot ID 232 and the broadcast bot ID 233 as target field values ​​in the target field of the data resource.

[0066] In some embodiments, the identification information can be read by a server-side interface of the information processing link to apply a target machine learning model to a target processing node. In such an embodiment, the server-side interface can read the identification information of the machine learning model corresponding to each of the at least one processing node from the target field of the data resource. In other words, the server-side interface can read all identification information of all machine learning models corresponding to the at least one processing node.

[0067] Thus, the server-side interface can replace the Bot ID originally used in the business logic with the newly configured Bot ID. In some embodiments, if the user 140 has not configured the model configuration information for a processing node in the at least one processing node, the Bot ID originally used in the business logic can be used for the processing node.

[0068] Through the above embodiment, corresponding operations are executed based on the data of the first processing node ranked first among at least one processing node, and the model configuration information corresponding to at least one processing node is uniformly read using a server-side interface. This enables the coordinated evaluation of multiple links, thereby resolving the issue of high use case construction costs. This approach is conducive to improving the efficiency of completing use case evaluations for serial links. The following describes the construction of the first evaluation use case for at least one processing node from the perspective of the use case execution platform 130.

[0069] 3A to 3C illustrate example interfaces 300A to 300C for executing use cases according to some embodiments of the present disclosure. In an embodiment of the present disclosure, interfaces 300A to 300C may be provided by the use case execution platform 130 shown in FIG1 .

[0070] 3A , the use case execution platform 130 can receive preset operations (e.g., click operations, long press operations, etc.) from the user 140 through the interface 300A to import the data object into the data set. If the data set is only used by one use case, it does not need to be entered into the data set here. The data set can be directly imported when creating the use case. Taking the data object as a CSV file as an example, the CSV file can be uploaded to the use case execution platform 130 through the button 312 on the interface 300A. After uploading, the user 140 can check whether the Bot ID at 314 on the interface 300A is the Bot ID configured on the user management platform 110, and can save the data set after checking that it is correct.

[0071] Referring to Figure 3B , a window 320 for creating a use case appears on the use case interface 300B of the use case execution platform 130. In window 320, the "Evaluation Object" field allows you to select a data object, and the "Secondary Object" field allows you to select the link to be tested. Referring to Figure 3C , on interface 300C of the use case execution platform 130, you can import a dataset and execute a use case. This facilitates subsequent use case evaluation on the user management platform 110.

[0072] In some embodiments, based on the data resource and at least one additional model configuration information different from the model configuration information, the use case management platform 110 can generate at least one additional data object. The at least one additional data object is used to construct at least one second evaluation case for at least one processing node. In such an embodiment, for the same data resource, each of the at least one processing node can correspond to different model configuration information in different evaluation cases.

[0073] As an example, each processing node can correspond to different types of machine learning models in different evaluation use cases. This allows for comparison of different types of machine learning models based on comparison reports, thereby selecting a more optimal type of machine learning model suitable for the data resource. As another example, each processing node can correspond to different versions of the same type of machine learning model in different evaluation use cases. This allows for comparison of different versions of the same type of machine learning model, thereby selecting a more optimal version of the machine learning model. It should be understood that this is merely an example and is not intended to be limiting.

[0074] In some embodiments, the use case management platform 110 may obtain a first evaluation report for a first evaluation case and at least one second evaluation report for at least one second evaluation case. Based on the received evaluation information, the use case management platform 110 may combine at least a portion of the first evaluation report with at least a portion of the at least one second evaluation report to form a comparison report. In some embodiments, at least a portion of the evaluation report may include at least model configuration information corresponding to the evaluation case. In other embodiments, at least a portion of the evaluation report may also include other information related to the evaluation report.

[0075] 2F , the use case management platform 110 can select the evaluation report to be compared and configure the evaluation information through interface 200F. For example, the user 140 can enter the name of the evaluation report for the evaluation case, select the evaluation link (e.g., broadcast link, summary link, etc.), select the evaluator, and select the number of evaluation reports to be compared. The evaluator can also select the number of samples to be evaluated. Referring again to FIG2G , the use case management platform 110 can add some or all of the evaluation information to the comparison report 250 for providing to the evaluator for scoring. The comparison report 250 can be a file in the Base document format.

[0076] In this embodiment, at least a portion of multiple evaluation reports for several evaluation use cases that require comparison can be combined into a single comparison report, facilitating comparison within a single report. This addresses the high cost of aggregating evaluation results in traditional solutions. Traditional solutions are unable to quickly generate a comparison document based on multiple evaluation reports. This embodiment facilitates the rapid generation of evaluation report comparisons and allows for configuration of user-focused metrics, making evaluation more flexible.

[0077] In some embodiments, in response to a preset operation, use case information for at least one evaluation use case is presented in a use case dimension. The preset operations discussed herein may, for example, include click operations (such as single-click operations, double-click operations, etc.), long press operations, sliding operations, keyboard triggering operations, and any other appropriate types of operations, which are not limited here. Referring to Figure 2H, on the use case maintenance interface 200H, use case information for multiple evaluation use cases may be presented. Use case information may, for example, include use case name, use case description, use case link, etc. Through the use case link on the use case maintenance interface 200H, you can jump directly to the corresponding use case, which is convenient and quick.

[0078] In summary, the disclosed embodiments enable the use case evaluation of a serial link using data from the first processing node and model configuration information from at least one processing node. This allows for quick and convenient data resource establishment, flexible completion of overall serial link evaluation, and improved efficiency in generating comparison reports, effectively saving costs.

[0079] Example Process

[0080] 4 shows a flow chart of an information processing link evaluation process 400 according to some embodiments of the present disclosure. Process 400 may be implemented at the use case management platform 110. Process 400 is described below with reference to FIG1.

[0081] In block 410 , the use case management platform 110 receives a data query request for a first processing node among one or more processing nodes included in an information processing link, where the one or more processing nodes are respectively configured to perform information processing using a machine learning model.

[0082] In block 420 , the use case management platform 110 generates a data resource for the first processing node based on the query result corresponding to the data query request. The data resource includes one or more data entries, each of which includes an input of the first processing node.

[0083] In block 430 , the use case management platform 110 receives model configuration information for at least one processing node to be evaluated among the one or more processing nodes, where the at least one processing node includes at least a first processing node.

[0084] In block 440 , the use case management platform 110 generates a data object based on the data resources and the model configuration information. The data object is used to construct a first evaluation case for at least one processing node.

[0085] In some embodiments, the at least one processing node includes a plurality of processing nodes arranged in sequence, and the first processing node is a first-ranked processing node among the plurality of processing nodes.

[0086] In some embodiments, the data query request indicates a query condition for a sample processed by the first processing node, and the query result is generated by: generating a query instruction corresponding to the query condition based on the query condition; and obtaining the query result by executing the query instruction.

[0087] In some embodiments, the query condition includes one or more constraints for the sample to be queried, and receiving the query condition includes: presenting corresponding setting entries for the one or more constraints; and receiving settings for the one or more constraints via the corresponding setting entries for the one or more constraints.

[0088] In some embodiments, the query condition includes one or more constraints on the samples to be queried, and the one or more constraints include at least one of the following: keywords included in the samples to be queried, the number of samples to be queried, or a time range corresponding to the samples to be queried.

[0089] In some embodiments, the model configuration information of the target processing node in at least one processing node includes identification information of the target machine learning model for the target processing node, and the identification information is used to be read by the server interface of the information processing link to use the target machine learning model for the target processing node.

[0090] In some embodiments, generating a data object based on the data resource and the model configuration information includes: adding identification information to the data resource as a target field value of a target field in the data resource; and generating the data object based on the data resource with the identification information added.

[0091] In some embodiments, process 400 also includes: generating at least one additional data object based on data resources and at least one additional model configuration information different from the model configuration information, and the at least one additional data object is used to construct at least one second evaluation case for at least one processing node; obtaining a first evaluation report for the first evaluation case and at least one second evaluation report for at least one second evaluation case; and based on the received evaluation information, merging at least a portion of the first evaluation report with at least a portion of the at least one second evaluation report into a comparison report.

[0092] In some embodiments, process 400 further includes: in response to a preset operation, presenting use case information for at least one evaluation use case in a use case dimension.

[0093] Example devices and equipment

[0094] 5 shows a schematic structural block diagram of an information processing link evaluation apparatus 500 according to certain embodiments of the present disclosure. Apparatus 500 may be implemented as or included in terminal device 110. Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0095] As shown in the figure, the device 500 includes a query request receiving module 510, which is configured to receive a data query request for a first processing node among one or more processing nodes included in the information processing link, and the one or more processing nodes are respectively configured to perform information processing using a machine learning model.

[0096] The apparatus 500 further includes a data resource generation module 520 configured to generate data resources for the first processing node based on a query result corresponding to the data query request, where the data resources include one or more data entries, each data entry including an input of the first processing node.

[0097] The apparatus 500 further includes a configuration information receiving module 530 configured to receive model configuration information for at least one processing node to be evaluated among the one or more processing nodes, where the at least one processing node includes at least a first processing node.

[0098] The apparatus 500 further includes a data object generation module 540 configured to generate a data object based on data resources and model configuration information, where the data object is used to construct a first evaluation case for at least one processing node.

[0099] In some embodiments, the at least one processing node includes a plurality of processing nodes arranged in sequence, and the first processing node is a first-ranked processing node among the plurality of processing nodes.

[0100] In some embodiments, the data query request indicates a query condition for the sample processed by the first processing node, and the device 500 also includes a query result generation module, which is configured to generate a query instruction corresponding to the query condition based on the query condition; and obtain the query result by executing the query instruction.

[0101] In some embodiments, the query condition includes one or more constraints for the sample to be queried, and the device 500 also includes a query condition receiving module, which is configured to present corresponding setting entries for one or more constraints; and receive settings for one or more constraints via the corresponding setting entries for one or more constraints.

[0102] In some embodiments, the query condition includes one or more constraints on the samples to be queried, and the one or more constraints include at least one of the following: keywords included in the samples to be queried, the number of samples to be queried, or a time range corresponding to the samples to be queried.

[0103] In some embodiments, the model configuration information of the target processing node in at least one processing node includes identification information of the target machine learning model for the target processing node, and the identification information is used to be read by the server interface of the information processing link to use the target machine learning model for the target processing node.

[0104] In some embodiments, the data object generation module 540 is further configured to add identification information to the data resource as a target field value of a target field in the data resource; and generate a data object based on the data resource with the identification information added.

[0105] In some embodiments, the device 500 also includes a comparison module, which is configured to generate at least one additional data object based on data resources and at least one additional model configuration information different from the model configuration information, and the at least one additional data object is used to construct at least one second evaluation case for at least one processing node; obtain a first evaluation report for the first evaluation case and at least one second evaluation report for at least one second evaluation case; and based on the received evaluation information, merge at least a portion of the first evaluation report with at least a portion of the at least one second evaluation report into a comparison report.

[0106] In some embodiments, the apparatus 500 further includes a use case information presentation module configured to present use case information for at least one use case in a use case dimension in response to a preset operation.

[0107] FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 can be used to implement the electronic device 110 of FIG1 .

[0108] As shown in FIG6 , electronic device 600 is a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 600.

[0109] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 600.

[0110] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0111] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0112] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0113] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0114] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0115] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0116] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0117] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0118] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for evaluating an information processing link, comprising: receiving a data query request for a first processing node among one or more processing nodes included in an information processing link, wherein the one or more processing nodes are respectively configured to perform information processing using a machine learning model; generating a data resource for the first processing node based on a query result corresponding to the data query request, the data resource comprising one or more data entries, each data entry comprising an input of the first processing node; receiving model configuration information for at least one processing node to be evaluated among the one or more processing nodes, the at least one processing node including at least the first processing node; as well as A data object is generated based on the data resource and the model configuration information, where the data object is used to construct a first evaluation case for the at least one processing node. 2 . The method of claim 1 , wherein the at least one processing node comprises a plurality of processing nodes arranged in sequence, and the first processing node is a first-ranked processing node among the plurality of processing nodes.

3. The method according to claim 1 , wherein the data query request indicates a query condition for the sample processed by the first processing node, and the query result is generated by: Based on the query condition, generating a query instruction corresponding to the query condition; and The query result is obtained by executing the query instruction.

4. The method according to claim 3, wherein the query condition includes one or more constraints on the sample to be queried, and Receiving the query condition includes: Presenting corresponding setting entries of the one or more constraints; as well as Settings for the one or more constraints are received via corresponding setting entries of the one or more constraints.

5. The method according to claim 3, wherein the query condition comprises one or more constraints on the sample to be queried, and the one or more constraints comprise at least one of the following: The keywords included in the sample to be queried, The number of samples to be queried, or The time range corresponding to the sample to be queried.

6. A method according to claim 1, wherein the model configuration information of the target processing node in the at least one processing node includes identification information of the target machine learning model for the target processing node, and the identification information is used to be read by the server-side interface of the information processing link to use the target machine learning model for the target processing node.

7. The method according to claim 6, wherein generating the data object based on the data resource and the model configuration information comprises: adding the identification information to the data resource as a target field value of a target field in the data resource; as well as The data object is generated based on the data resource to which the identification information is added.

8. The method according to claim 1, further comprising: generating at least one additional data object based on the data resource and at least one additional model configuration information different from the model configuration information, wherein the at least one additional data object is respectively used to construct at least one second evaluation case for the at least one processing node; Obtaining a first evaluation report for the first evaluation case and at least one second evaluation report for the at least one second evaluation case; as well as Based on the received evaluation information, at least a portion of the first evaluation report and at least a portion of the at least one second evaluation report are combined into a comparison report.

9. The method according to claim 1, further comprising: In response to a preset operation, use case information for at least one use case is presented in a use case dimension.

10. An information processing link evaluation device, comprising: a query request receiving module configured to receive a data query request for a first processing node among one or more processing nodes included in the information processing link, the one or more processing nodes being respectively configured to perform information processing using a machine learning model; a data resource generation module configured to generate a data resource for the first processing node based on a query result corresponding to the data query request, the data resource comprising one or more data entries, each data entry comprising an input of the first processing node; a configuration information receiving module configured to receive model configuration information for at least one processing node to be evaluated among the one or more processing nodes, the at least one processing node including at least the first processing node; as well as The data object generation module is configured to generate a data object based on the data resource and the model configuration information, wherein the data object is used to construct a first evaluation case for the at least one processing node.

11. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images