Application debugging method, information query method, and application evaluation method

Abstract

Provided in the embodiments of the present disclosure are an application debugging method and apparatus, an information query method and apparatus, an application evaluation method and apparatus, and a device and a storage medium. The application debugging method comprises: executing a test query for a target application; presenting corresponding first operation information of a plurality of stages for executing the test query, wherein the operation information of the plurality of stages at least indicates respective outputs of the plurality of stages; and generating an operation reference case for the target application on the basis of the corresponding first operation information of the plurality of stages for the test query, wherein the operation reference case comprises the test query and the operation information of the plurality of stages, and the operation reference case is referenced for determining a result of a user query during the operation of the target application. In this way, an application-creating user can improve the quality of services provided by a target application without the need for modifying code, thereby simplifying the debugging process of the target application.

Description

Application debugging methods, information query methods, and application evaluation methods

[0001] This application claims priority to the Chinese invention patent application entitled “Method for application debugging, method for information query, method for application evaluation” filed on November 22, 2023, with application number 202311570383.1, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to a method for application debugging, a method for information querying, a method for application evaluation, an apparatus, a device, and a computer-readable storage medium. Background Art

[0003] With the development of information technology, various terminal devices can provide people with a variety of services in work and life. Applications that provide these services can be deployed on these devices. These applications present content and interact with users through their user interfaces, meeting their needs. In some cases, users may initiate queries within applications. Therefore, improving the quality of query services provided to users is a key concern.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for application debugging is provided. The method includes: executing a test query for a target application; presenting corresponding first operation information for multiple stages of executing the test query, wherein the operation information for the multiple stages at least indicates outputs of each of the multiple stages; and generating an operation reference case for the target application based on the corresponding first operation information for the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information for the multiple stages, and the operation reference case is used as a reference to determine the result of the user query during the operation of the target application.

[0006] In a second aspect of the present disclosure, a method for information query is provided. The method includes: in response to receiving a user query for a target application, determining, based on a similarity between the user query and a test query exceeding a threshold similarity, a running reference case including the test query as a target running reference case for the user query, wherein the running reference case includes the test query and corresponding running information for multiple stages of the test query; and executing the user query based on the target running reference case.

[0007] In a third aspect of the present disclosure, a method for application evaluation is provided. The method includes: generating an evaluation reference case for a target application, the evaluation reference case including an evaluation query, a first result generated by the target application in response to the evaluation query, and evaluation feedback on the first result; selecting a target evaluation sample from the evaluation sample set based on similarity between the evaluation query and queries in an evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application in response to the sample query; and generating an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0008] In a fourth aspect of the present disclosure, a device for application debugging is provided. The device includes: a test query execution module configured to execute a test query for a target application; a first operation information presentation module configured to present corresponding first operation information of multiple stages of executing the test query, wherein the operation information of the multiple stages at least indicates the output of each of the multiple stages; and an operation reference case generation module configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced during the operation of the target application to determine the result of the user query.

[0009] In a fifth aspect of the present disclosure, a device for information query is provided. The device includes: an operation reference case determination module configured to, in response to receiving a user query for a target application, determine an operation reference case including a test query as a target operation reference case for the user query based on a similarity between the user query and a test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; and a user query execution module configured to execute the user query based on the target operation reference case.

[0010] In a sixth aspect of the present disclosure, a device for application evaluation is provided. The device includes: an evaluation reference case generation module configured to generate an evaluation reference case for the target application, the evaluation reference case including an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; a target evaluation sample selection module configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; and an evaluation result generation module configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0011] In a seventh aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the methods of the first and second aspects.

[0012] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect and the method of the second aspect are implemented.

[0013] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0015] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0016] FIG2 shows an example of a debugging entry page for each function in a target application according to some embodiments of the present disclosure;

[0017] FIG3 shows an example of a page of test query records for a specific function in a target application according to some embodiments of the present disclosure;

[0018] FIG4 shows an example of a page for debugging a target application according to some embodiments of the present disclosure;

[0019] FIG5 shows an example of a page displaying data information according to some embodiments of the present disclosure;

[0020] FIG6A shows an example of a page of a use case library for a specific function in a target application according to some embodiments of the present disclosure;

[0021] FIG6B shows an example of a use case details page according to some embodiments of the present disclosure;

[0022] FIG7A shows a flowchart of a process for application debugging according to some embodiments of the present disclosure;

[0023] FIG7B shows a flowchart of an information query process according to some embodiments of the present disclosure;

[0024] FIG7C shows a flowchart of a process for application evaluation according to some embodiments of the present disclosure;

[0025] FIG8A shows a schematic structural block diagram of an apparatus for application debugging according to some embodiments of the present disclosure;

[0026] FIG8B shows a schematic structural block diagram of an apparatus for information query according to some embodiments of the present disclosure;

[0027] FIG8C shows a schematic structural block diagram of an apparatus for application evaluation according to some embodiments of the present disclosure; and

[0028] FIG9 shows a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0029] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0030] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below.

[0031] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and authorization should be obtained from relevant users in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to the software or hardware such as the electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message may be sent to the relevant user in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0036] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0037] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions. The activation of the relevant functions of the digital assistant in this disclosed embodiment, the data acquired, the processing and storage of the data, etc., shall all be authorized in advance by the user and other rights holders associated with the user, and shall comply with the provisions of relevant laws, regulations and agreement rules between rights holders.

[0038] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0039] FIG1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Environment 100 includes an application management platform 110, which can support the creation and / or execution of applications. In some embodiments, the portion of application management platform 110 that supports application creation can also be referred to as an application creation portion. In some embodiments, the portion of application management platform 110 that supports application execution can also be referred to as an application execution portion.

[0040] As shown in Figure 1, the application creation part can provide a user 105 with an environment for creating and publishing applications. User 105 can be referred to as an application creation user or creator. In some embodiments, the application creation part can be a low-code platform that provides a collection of tools for application creation. The application creation part can support visual development of various types of applications, allowing developers to skip the manual coding process and speed up the application development cycle and cost. The application creation part can support any appropriate platform for users to develop one or more types of applications, for example, it can include a platform based on Application Platform as a Service (aPaaS). Such a platform can support users to efficiently develop applications and implement operations such as application creation and application function adjustment.

[0041] The application creation part can be deployed locally on the terminal device of user 105, and / or can be supported by a server-side device. For example, the terminal device of user 105 can run a client of the application creation part, which can support the interaction between the user and the application creation part provided by the server. In the case where the application creation part runs locally on the user's terminal device, user 105 can directly use the terminal device to interact with the local application creation part. In the case where the application creation part runs on the server-side device, the server-side device can realize the service provision of the client running in the terminal device based on the communication connection between the terminal device. The application creation part can present a corresponding page 130 to the user 105 based on the operation of the user 105 to output and / or receive information related to application creation to the user 105 and / or from the user 105.

[0042] In some embodiments, the application creation portion can be associated with a corresponding database, which stores the data or information required for the application creation process supported by the application creation portion. For example, the database can store the code and description information corresponding to each functional module used to make up the application. The application creation portion can also perform operations such as calling, adding, deleting, and updating the functional modules in the database. The database can also store operations that can be performed on different functional blocks. For example, in a scenario where an application is to be created, the application creation portion can call the corresponding functional blocks from the database to build the application.

[0043] In an embodiment of the present disclosure, a user 105 can create a target application 120 as needed in the application creation section and publish the target application 120. The target application 120 can be published to any appropriate application execution section, as long as the application execution section can support the operation of the target application 120. After publication, the target application 120 can be used to be operated by one or more users 145. The user 145 can be referred to as the end user of the target application 120. In some embodiments, the target application 120 may include or be implemented as a digital assistant 122.

[0044] The digital assistant 122 can be configured to have intelligent dialogue. In the example shown in Figure 1, the digital assistant 122 can be integrated into the target application 120 and assist in executing task processing within the target application 120 as a part of the target application 120. In other examples, the digital assistant 122 can be configured as an independently running application, such as a web application or other types of applications. In such an example, the digital assistant 122 and the target application 120 can be regarded as the same application. The digital assistant 122 is provided to assist users with various task processing needs in different applications and scenarios. During the interaction with the digital assistant 122, the user inputs an interactive message, and the digital assistant 122 provides a reply message in response to the user input. Generally, the digital assistant 122 can support users to input questions in natural language, and perform tasks and provide replies based on the understanding of natural language input and logical reasoning ability.

[0045] In some embodiments, digital assistant 122 can interact with user 145 as a contact. For example, digital assistant 122 can be implemented in an instant messaging (IM) application. Digital assistant 122 can interact with user 145 in a single chat session with user 145. In some embodiments, digital assistant 122 can interact with multiple users in a group chat session including multiple users.

[0046] For each user 145, the client of the application running portion can present an interaction window 142 of the target application 120 or the digital assistant 122 in the client interface, such as a conversation window with the digital assistant 122. The user 145 can enter a conversation message in the conversation window, and the target application 120 can determine a reply message of the digital assistant 122 based on the created configuration information and present it to the user in the interaction window 142. In some embodiments, depending on the configuration of the target application 120, the interaction message with the target application 120 can include messages in multimodal forms, such as text messages (e.g., natural language text), voice messages, image messages, video messages, etc.

[0047] Similar to the application creation part, the application running part can be deployed locally on the terminal device of each user 145, and / or can be supported by a server-side device. For example, the terminal device of user 145 can run a client with the application running part, which can support the interaction between the user and the application running part provided by the server. In the case where the application running part runs locally on the user's terminal device, the user 145 can directly use the terminal device to interact with the local application running part. In the case where the application running part runs on the server-side device, the server-side device can realize the service provision of the client running in the terminal device based on the communication connection between the terminal device. The application running part can present the corresponding application page to the user 145 based on the operation of the user 145, so as to output and / or receive information related to the use of the application to the user 145 and / or receive information from the user 145 from the user 145.

[0048] In some embodiments, at least some of the functionality of the target application 120 and / or at least some of the functionality of the digital assistant 122 in the target application 120 can be implemented based on a model. During the creation or execution of the target application 120, one or more models 155, such as the capabilities of the models 155, can be invoked. In the target application 120, the digital assistant 122 can utilize the models 155 to understand user input and provide responses to the user based on the output of the models 155.

[0049] During the creation process, the application creation platform 110 tests the target application 120 using the model 155 to ensure that the target application 120's operating results meet expectations. During operation, in response to various user requests for the target application 120, the application execution component may need to use the model 155 to determine the user's response.

[0050] Although shown as being independent of the application management platform 110, one or more models 155 can run on the application management platform 110 or other remote servers. In some embodiments, the model 155 can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model can be based on a language model (LM). A language model can have question-answering capabilities by learning from a large amount of corpus. The model 155 can also be based on other appropriate models.

[0051] The management platform 110 can run on appropriate electronic devices. The electronic devices here can be any type of device with computing capabilities, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can, for example, include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and so on. In some embodiments, the management platform 110 can be implemented based on cloud services.

[0052] It should be understood that the structure and functionality of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. For example, although FIG1 illustrates a single user interacting with the application creation portion and a single user interacting with the application execution portion, in practice multiple users may access the application management platform 110 to each create a digital assistant, and each digital assistant may be used to interact with multiple users.

[0053] When a user initiates a query request within an app, the app can leverage machine learning models to provide query services. Initially, the quality of the query service provided by the app may be low, requiring multiple rounds of optimization to improve its effectiveness. Traditional optimization requires system improvements at the code level, which is costly, time-consuming, and prone to errors.

[0054] To this end, an application debugging solution is provided according to an embodiment of the present disclosure. According to various embodiments of the present disclosure, a test query is executed for a target application. Corresponding first operation information of multiple stages of executing the test query is presented, and the operation information of the multiple stages at least indicates the output of each of the multiple stages. Based on the corresponding first operation information of the multiple stages of the test query, an operation reference case for the target application is generated. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced during the operation of the target application to determine the result of the user query.

[0055] In various embodiments of the present disclosure, application creators can generate a reference run case for the target application based on the corresponding operational information from multiple stages of executing a test query. This reference run case can then be used as a reference when generating query results for user queries initiated by end users. In this way, application creators can improve the quality of services provided by the target application without modifying the code, simplifying the target application tuning process.

[0056] Some example embodiments of the present disclosure will be described in detail below with reference to the examples in the accompanying drawings. It should be understood that the pages shown in the accompanying drawings are merely examples, and a variety of page designs may exist. The various graphical elements in a page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present. The embodiments of the present disclosure are not limited in this respect.

[0057] The following describes example embodiments of the present disclosure with reference to the accompanying drawings. For discussion purposes, the following examples are described from the perspective of an application management platform, such as application management platform 110 shown in FIG1 . The pages presented by application management platform 110 can be presented via a terminal device of user 105 , and user input can be received via the terminal device of user 105 .

[0058] Figure 2 illustrates an example of a page 200 showing debugging entry points for various functions within a target application 120 according to some embodiments of the present disclosure. As shown in Figure 2 , a page 200 showing debugging entry points for various functions within the target application 120 is presented on the terminal device of user 105. Page 200 provides a functional area 210, which can be used for various functions within the target application. Functions within the target application may include, for example, application data analysis, application data updates, and knowledge question-and-answer functions. However, it should be understood that these are merely exemplary functions and are not intended to be limiting. The target application herein can be any suitable application, such as a human resources application, a CRM (Customer Relationship Management) application, or a financial application. Taking the debugging of the application data analysis function within the target application as an example, user 105 triggers a debug control 220 for the data analysis function, for example, by clicking on the debug control 220. In response, a page showing test query records may be presented to user 105.

[0059] Figure 3 shows an example of a page 300 recording test queries for a specific function in a target application according to some embodiments of the present disclosure. As shown in Figure 3, page 300 recording test queries for a specific function in a target application can be presented on user 105's terminal device. Page 300 records test queries previously created by user 105, and user 105 debugs the corresponding function by entering the test query. In some embodiments, user 105 can re-debug a previously created test query. For example, user 105 can click control 310 to re-execute the test query "What is the quantity of products with a unit price less than XX yuan?" to debug the corresponding function in the target application. In response, a debugging page can be presented to user 105. In some embodiments, user 105 can trigger the execution of a new test query. For example, user 105 can click the newly added control 320 to enter a new test query to debug the corresponding function in the target application. In response, user 105 can enter the debugging page.

[0060] FIG4 shows an example of a page 400 for debugging a target application according to some embodiments of the present disclosure. The application management platform 110 can execute a test query for the target application. Referring to FIG4 , for example, the user 105 enters a test query statement in the text area 410 corresponding to the test query and clicks the control 420 to execute a test query for the target application. The test query statement entered by the user 105 can be, for example, "What is the quantity of goods with a unit price lower than XX yuan?", "What is the proportion of goods with an origin of XX? Draw a pie chart". The application management platform 110 can present the corresponding first operation information of multiple stages of executing the test query in the staged debugging area 430, and the operation information of the multiple stages at least indicates the output of each of the multiple stages.

[0061] After executing a test query against a target application, application management platform 110 may present corresponding first execution information for multiple stages of executing the test query in staged debugging area 430. The execution information for each stage at least indicates the output of each stage. For example, staged debugging area 430 displays execution information for five stages, each requiring the execution of a task. After the tasks are completed, corresponding outputs 440, 442, 444, 446, and 448 are generated. For example, output 444 of the third stage may include the data tables required to execute the test query.

[0062] In some embodiments, the operation information of multiple stages may further indicate at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, and the operation status of each of the multiple stages (i.e., whether the operation is completed). In some embodiments, due to the limitations on the input information of the machine learning model (e.g., the length of the input information), the operation information of multiple stages may further indicate the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages. In this way, user 105 can make the input information provided to the machine learning model meet the requirements of the machine learning model, thereby ensuring the stable operation of the machine learning model.

[0063] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include at least one predefined stage, and at least one predefined stage includes a stage for generating a database query request. The at least one predefined stage may be some debugging stages pre-set by the application management platform 110 according to the characteristics and / or corresponding functions of the target application. The purpose of the application management platform 110 performing a test query for the target application may be to generate a better database query request to return more accurate query results to the user. Therefore, the at least one predefined stage may also include a stage for generating a database query request, which may generate an accurate and efficient database query request based on the output of the previous predefined stage. The database query request here may be, for example, an SQL (Structured Query Language) statement.

[0064] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include, in addition to at least one predefined stage, at least one stage determined based on a test query. After the application management platform 110 executes a database query request and obtains the corresponding data, it may also determine at least one stage based on the requirements of the test query. In one example, the test query is "What is the number of products with a unit price less than XX yuan?" This test query does not require additional data analysis, so the application management platform 110 may only determine one stage, such as the data review stage, in which the application management platform 110 may output the number of products with a unit price less than XX yuan. In another example, the test query is "What is the percentage of products produced in XX? Draw a pie chart?" This test query requires additional data analysis, so the application management platform 110 may generate multiple stages, such as a chart generation stage and a data insight stage. The application management platform 110 may output a pie chart showing the percentage of products produced in XX to the chart generation stage. The application management platform 110 may also output data mining and analysis on the percentage of products produced in XX to the data insight stage. In this way, the application management platform 110 can determine different stages after the predefined stage based on the test query, which can meet different query requirements of users and improve the query experience of users.

[0065] In some embodiments, the test query executed by the application management platform 110 for the target application is a query for the first function in the target application. Because different functions in the target application have significant differences, debugging one function in the target application may not be applicable to another function in the target application. Therefore, the test query is a query for a specific function in the target application.

[0066] In some embodiments, the predefined phases may differ for different functions in the target application. For example, taking the first function as an example, at least one predefined phase may be configured for the first function in the target application. The application management platform 110 may configure at least one different predefined phase based on the characteristics of different functions in the target application. For example, for the application data analysis function in the target application, the application management platform 110 may configure five predefined phases: recall and select terms, recall and select reference use cases, recall and select target datasets, recall and select fields belonging to the target dataset, and generate and execute database query requests. The recall and select terms phase may recall and select interpretations of terms used in the test query. The recall and select reference use cases phase may recall and select reference use cases similar to the test query. The recall and select target dataset phase may recall and select the dataset required for executing the test query. The recall and select fields belonging to the target dataset phase may recall and select the fields within the dataset required for executing the test query. The generate and execute database query request phase may generate and execute a database query request based on the outputs of the previous four phases.

[0067] The above describes the phase division using the data analysis function as an example, but this division is merely exemplary. Furthermore, for other functions (e.g., data updates, knowledge questions, etc.), appropriate phase divisions can be made based on the characteristics of the corresponding functions. In this way, different functions in the target application can be debugged separately using test queries, fully considering the differences between different functions and effectively improving the service quality provided by different functions.

[0068] After presenting the corresponding first operation information of multiple stages of executing the test query, the application management platform 110 can generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of multiple stages. The operation reference case is used as a reference to determine the result of the user query during the operation of the target application. For example, the user 105 can first click the tuned control 450 on the page 400 to confirm that the test query and the corresponding operation information have been tuned, and then click the save control 460 to generate the operation reference case. After the operation reference case is generated, the application management platform 110 can refer to the operation reference case to determine the result of the user query when receiving the user query of the user 145.

[0069] In some embodiments, the application management platform 110 can determine adjustments to the first operating information of at least one of the multiple stages based on the corresponding first operating information of the multiple stages of the test query. The application management platform 110 can then re-execute the test query based on the adjusted first operating information of at least one stage. The application management platform 110 can present the corresponding second operating information of the multiple stages of the re-executed test query. If a use case generation confirmation is received, the application management platform 110 can generate a reference use case for execution based on the corresponding second operating information. After the first operating information of at least one stage is adjusted, the application management platform 110 can re-execute the test query and present the corresponding second operating information of the multiple stages of the re-executed test query in the staged debugging area 430. Referring to Figure 4, the application management platform 110 can receive the use case generation confirmation through the user 105 triggering the tuned control 450 and the save control 460 to generate a reference use case for execution based on the second operating information.

[0070] In some embodiments, the application management platform 110 may receive terminology adjustment information from the user 105 for the first stage of multiple stages, and re-execute the test query based on the terminology adjustment information. The terminology adjustment information indicates adding a term set to the first stage or removing a term set selected in the first stage. The term set includes explanations of the terms used by the user group of the target application. The terms used by the user group may be used within the scope of the group, but are not common terms, so it is necessary to help the machine learning model understand such terms. The explanation of such a term may include a literal explanation of the term, and may also include a common term corresponding to the term.

[0071] Exemplarily, the first stage of executing the test query may select a term set related to the test query, and the user 105 may add or remove a term set in the first stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods sold through the C2C channel?", where C2C is a specialized term. The term set selected in the first stage may include an explanation of C2C, and the user 105 may add other term sets or delete the term set selected in the first stage. After adjusting the term set, the application management platform 110 may determine one or more term sets used in the first stage based on the term adjustment information, and then re-execute the test query based on the one or more term sets.

[0072] In some embodiments, the application management platform 110 can receive use case adjustment information for the second phase of multiple phases and re-execute the test query based on the use case adjustment information. The use case adjustment information indicates the addition of reference use cases to the second phase or the removal of reference use cases selected in the second phase. For example, the second phase of executing the test query can select use cases similar to the test query, and the user 105 can add or remove reference use cases in the second phase. For example, if the user 105's test query for the target application is "How many products have a unit price less than 5,000 yuan?", the second phase can select reference use cases similar to the test query based on similarity calculation. The reference use case selected in the second phase can be "How many products have a unit price less than 8,000 yuan?" The user 105 can add other reference use cases or move the reference use cases selected in the second phase. After adjusting the reference use cases, the application management platform 110 can determine one or more reference use cases to be used in the second phase based on the use case adjustment information, and then re-execute the test query based on the one or more reference use cases.

[0073] In some embodiments, the application management platform 110 may receive data set adjustment information for the third stage of the multiple stages and re-execute the test query based on the data set adjustment information. For example, the third stage of executing the test query may select the data set required to execute the test query, and the user 105 may add or remove data sets in the third stage. For example, the test query of the user 105 for the target application is "What is the number of products with a unit price of less than 5,000 yuan?" The data set selected in the third stage may be a product analysis table. The user 105 may add other data sets or delete the data set selected in the third stage.

[0074] In some embodiments, the application management platform 110 can also receive data set field adjustment information for the fourth stage among the multiple stages, and re-execute the test query based on the data set field adjustment information. Exemplarily, the fourth stage of executing the test query can select the fields of the data set required to execute the test query, and the user 105 can add fields or remove fields in the fourth stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods with a unit price of less than 5,000 yuan?", and the fields of the data set selected in the fourth stage are the product number field, unit price field, origin field and production date field in the product analysis table. The user 105 can add other fields or delete the fields selected in the fourth stage. In this way, the user can adjust the various stages of the test query through a graphical interface, which facilitates the user's understanding of debugging and simplifies the debugging process of the target application.

[0075] In some embodiments, the application management platform 110 may receive a stage operation request after determining the adjustment to the first operation information of at least one stage. The stage operation request indicates that a test query is to be executed from a specified stage among multiple stages, where the specified stage is before at least one stage or is included in at least one stage, and based on the adjusted first operation information of at least one stage, the specified stage among the multiple stages and the stage after the specified stage are re-executed. Exemplarily, the user 105 adjusts the second stage, and the specified stage may be the first stage before the second stage, or the second stage. In the case where the specified stage is the first stage, the application management platform 110 may execute the first stage and the stage after the first stage (i.e., all stages) based on the adjusted first operation information of at least the second stage.

[0076] For example, referring to Figure 4 , a trigger entry for re-execution of each of the five stages is provided, such as trigger controls 421, 422, 423, 424, and 425. User 105 can trigger control 420 to restart all stages from the beginning. If the designated stage is the second stage, the application management platform 110 can execute the second stage and subsequent stages based on the adjusted first operation information of at least the second stage. For example, continuing with Figure 4 , user 105 can trigger control 422 to execute the second stage and subsequent stages. In this way, users can flexibly execute different stages and verify the effectiveness of adjustments made to different stages by comparing the output results of running different stages. It should be understood that although Figure 4 shows the execution of the corresponding stage through the triggering of controls 421, 422, 423, 424, and 425, this triggering method is merely illustrative. User 105 can also trigger the corresponding stage through other methods, such as through user input representing natural language to trigger the execution of the corresponding stage.

[0077] In some embodiments, after re-executing the test query, the application management platform 110 may present operation information associated with the re-execution of the specified stage and the stages subsequent to the specified stage in the staged debugging area 430. For example, if the specified stage is the second stage, operation information associated with the re-execution of the second stage and the stages subsequent to the second stage may be presented in the staged debugging area 430.

[0078] In some embodiments, the application management platform 110 may also present data information simultaneously with the corresponding first operation information for the multiple stages, where the data information at least indicates at least one data set used to execute the test query, and generate an operation reference case based on the corresponding first operation information for the multiple stages and the data information. Referring to Figure 4 , the application management platform 110 may display the data information used to execute the test query in data information display area 470 . Furthermore, the application management platform 110 may generate an operation reference case based on the data information or adjustments to the data information.

[0079] In some embodiments, adjusting the data information may include updating at least one dataset used to execute the test query. For example, a dataset used to execute the test query, namely a product analysis table, is displayed in data information display area 470. Referring to FIG. 4 , user 105 may trigger add control 472 to update the dataset.

[0080] In some embodiments, adjusting the data information may include updating at least a portion of at least one data set used to perform the test query. Here, the portion of the data set may indicate a field description within the data set. For example, data information display area 470 displays a data set used to perform the test query, namely, a product analysis table. Referring to FIG. 4 , user 105 may trigger modification control 474 to update the field description within the data set.

[0081] In some embodiments, the data information also indicates a terminology set referenced for executing the test query, which includes interpretations of terms used by the user community of the target application. Referring to FIG5 , FIG5 illustrates an example of a page 500 displaying data information according to some embodiments of the present disclosure. Page 500 includes a terminology set display area 510 , and user 105 can adjust the content of terminology set display area 510 .

[0082] In some embodiments, the data information further indicates the conversation record between the user and the digital assistant referenced for executing the test query. Continuing with reference to FIG5 , page 500 includes a conversation record display area 520 , and user 105 can adjust the content in the conversation record display area 520 .

[0083] In some embodiments, the data information also indicates reference prompts used to execute the test query. Continuing with FIG5 , page 500 includes a reference prompt display area 530 , where user 105 can adjust the content of reference prompt display area 530 . By adjusting the reference prompts, the machine learning model can be better guided, helping it to better complete the user's task.

[0084] In some embodiments, the data information further indicates conditions for determining candidate results for the test query. Continuing with FIG5 , page 500 includes a condition area 540 , in which user 105 can adjust the similarity and number of search terms. For example, in a knowledge quiz scenario, the digital assistant searches for questions similar to the user's based on similarity and provides a predetermined number of search results to user 145 based on the number of searches per session.

[0085] Referring to FIG6A , FIG6A illustrates an example of a page 600 for a use case library for a specific function in a target application according to some embodiments of the present disclosure. Page 600 presents a use case library containing generated reference use cases. User 105 triggers detail control 602 to jump to a page for use case details corresponding to the use case, as shown in FIG6B . FIG6B illustrates an example of a page 610 for use case details according to some embodiments of the present disclosure. Page 610 may include test queries and corresponding first run information 614 for multiple stages.

[0086] It can be understood that, according to the embodiments described above, an operation reference use case set including one or more operation reference use cases can be generated.

[0087] In some embodiments, after the target application 120 is published, during the operation of the target application 120, a reference run case can be utilized to provide services to the user 145. For example, in response to receiving a user query for the target application, the application management platform 110 can search for a test query similar to or matching the user query from a collection of reference run cases. Based on the similarity between the user query and a test query exceeding a threshold similarity, the application management platform 110 can determine the reference run case as the target reference run case for the user query and execute the user query based on the target reference run case. After generating the reference run case, the application management platform 110 receives a user query for the target application initiated by the user 145. The application management platform 110 can determine the similarity between the user query and the test query through similarity calculation and, based on the similarity, determine the reference run case as the target reference run case for the user query. After determining the target reference run case, the application management platform 110 can use the target reference run case to execute the user query. In this way, by referencing the reference run case to determine the results of the user query, more accurate query results can be provided to the user, thereby improving user satisfaction.

[0088] In some embodiments, a machine learning model can be used to execute user queries. For example, the application management platform 110 can provide the determined target reference use case to the machine learning model as part of the prompt word for the user query. The application management platform 110 can then obtain the execution result of the user query from the machine learning model. It should be noted that the target reference use case here is selected from the set of running reference use cases generated during the application creation phase or the application testing phase. Such target reference use cases are generated for debugging purposes, rather than from use cases when users actually use the application.

[0089] In some embodiments, the user using target application 120 may be a tester, meaning that the user query may originate from a tester. In this case, if the tester provides positive feedback on a query result, a reference run case can be generated based on the user query and the corresponding query result. For example, operational information from multiple stages of the operational process leading to the query result can be stored as a reference run case. In this way, user feedback is leveraged to enrich the source of reference run cases.

[0090] In some embodiments, the target application includes at least one function, and both the test query and the user query are queries targeting a first function of the at least one function. The target application may include functions such as application data analysis, application data update, and knowledge question answering. The test query and the user query need to target the same function. For example, the test query and the user query may both target the application data analysis function.

[0091] During creation or testing of the target application 120, it may be necessary to evaluate the performance of the target application 120. For example, in the case where the target application includes multiple functions, it may be necessary to evaluate one or some of the functions.

[0092] To this end, in some embodiments, the application management platform 110 may maintain a set of evaluation samples for use in evaluating the target application 120, for example, evaluating one or more functions of the target application 120, such as a data analysis function, a data update function, a knowledge question and answer function, etc. Each evaluation sample in the evaluation sample set may include a sample query and a result (also referred to as a sample query result or a second result) generated by the target application 120 in response to the sample query.

[0093] In some embodiments, to obtain evaluation results for a function in the target application 120, a machine learning model can be used to obtain feedback on sample query results in the evaluation sample. For example, the machine learning model can score the sample query results generated by the target application 120 based on one or more dimensions.

[0094] The machine learning model can give feedback in any suitable manner. As an example, an automatic evaluation method is described. The data source (e.g., knowledge base, data set, etc.) used by the target application 120 to provide services can be provided to the machine learning model, so that the machine learning model extracts queries and corresponding query results therefrom as correct examples. For example, for the knowledge question and answer function, questions and corresponding answers can be extracted. Then, the extracted queries can be executed in the target application 120, and the execution results are generated by the target application 120. Furthermore, the machine learning model can give feedback on the execution results based on the answers and execution results in various suitable dimensions, such as scoring. Examples of evaluation dimensions may include but are not limited to helpfulness to users, language fluency, factual correctness, logic, etc.

[0095] Thus, the machine learning model provides feedback based on the extracted questions and execution results to determine the evaluation results for the target application 120. For example, if the evaluation sample set is for a knowledge question and answer function, the evaluation results for that function can be obtained. In this way, the user 105 can debug the target application 120 based on the evaluation results, for example, debugging one or more functions.

[0096] During the creation or testing of target application 120, the creator or tester may enter some evaluation queries. Target application 120 can generate corresponding query results for the evaluation queries, also known as first results or evaluation query results. The creator or tester can then provide feedback on the evaluation query results, also known as evaluation feedback.

[0097] In some embodiments, evaluation feedback can be used in conjunction with the evaluation sample set to provide more accurate evaluation results for the target application 120. For example, the operation management platform 110 can generate an evaluation reference case for the target application 120 based on the evaluation query provided by the creator or tester. The evaluation reference case includes the evaluation query from the creator or tester, the evaluation query result generated by the target application 120 in response to the evaluation query, and evaluation feedback on the evaluation query result. The evaluation feedback can be positive or negative, and can also indicate the user 145's satisfaction with the query result.

[0098] Generally speaking, for similar queries, it is expected that the evaluation results and evaluation feedback are consistent or matching. In view of this, sample queries similar to the evaluation query can be found from the evaluation sample set, for example, with a similarity exceeding a threshold similarity. The evaluation sample including such similar sample queries is also referred to as the target evaluation sample, which also includes the sample results generated by the target application 120 for the sample query.

[0099] The application management platform 110 can generate evaluation results for the target application 120 based on the evaluation query results, evaluation feedback, and sample query results. For example, the scores of the sample query results in the target evaluation sample can be matched with the evaluation feedback. It is understood that the evaluation reference use cases provide examples of what results satisfy the creator or tester and what results dissatisfy the creator or tester.

[0100] In some embodiments, as described above, a machine learning model can be used to perform an evaluation of the target application 120. Accordingly, a prompt word can be generated based on at least a portion of the evaluation reference case (e.g., evaluation feedback and evaluation query results), and the prompt word can be provided to the machine learning model. The machine learning model can generate feedback on the sample query results in the target evaluation sample. For example, the machine learning model can score the sample query results. Based on the feedback, the evaluation results for the target application 120 can then be determined. It can be understood that the evaluation reference case provides examples of what kind of results the creator or tester is satisfied with or what kind of results the creator or tester is dissatisfied with. In this way, the machine learning model can refer to such examples when generating feedback (e.g., scoring) for the sample results. Note that the machine learning model used in the evaluation can be the same as or different from the machine learning model used in the operation of the target application 120.

[0101] In some embodiments, if the target application 120 includes at least one function, the sample queries used for evaluation and the evaluation queries may be queries targeting the same function. For example, the evaluation sample set may be dedicated to evaluating the knowledge question and answer function of the target application 120. In this case, the collected user queries are also targeted at the knowledge question and answer function.

[0102] FIG7A shows a flowchart of a process 700 for application debugging according to some embodiments of the present disclosure. The process 700 may be implemented at the application management platform 110. The process 700 is described below with reference to FIG7.

[0103] In block 701 , the application management platform 110 executes a test query for a target application.

[0104] In block 702 , the application management platform 110 presents corresponding first running information of multiple stages of executing the test query, where the running information of the multiple stages at least indicates outputs of the respective stages.

[0105] In box 703, the application management platform 110 generates an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0106] In some embodiments, generating an operation reference use case includes: determining an adjustment to first operation information of at least one stage among multiple stages; re-executing a test query based on the adjusted first operation information of at least one stage; presenting corresponding second operation information of multiple stages of the re-executed test query; and generating an operation reference use case based on the corresponding second operation information in response to receiving a use case generation confirmation.

[0107] In some embodiments, re-executing the test query includes: receiving a stage running request, the stage running request instructing to execute the test query from a specified stage among multiple stages, the specified stage being before at least one stage or being included in at least one stage; and re-executing the specified stage among the multiple stages and the stages after the specified stage based on the adjusted first running information of at least one stage.

[0108] In some embodiments, presenting corresponding second execution information of the plurality of stages of re-executing the test query includes presenting execution information associated with re-execution of a specified stage and stages subsequent to the specified stage.

[0109] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0110] In some embodiments, determining an adjustment to first run information of at least one of a plurality of stages includes: receiving term adjustment information for a first stage of a plurality of stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and re-executing the test query includes: determining one or more term sets used in the first stage based on the term adjustment information; and re-executing the test query based on the one or more term sets.

[0111] In some embodiments, determining an adjustment to first operating information of at least one stage among multiple stages includes: receiving use case adjustment information for a second stage among the multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and re-executing the test query includes: determining one or more reference use cases used in the second stage based on the use case adjustment information; and re-executing the test query based on the one or more reference use cases.

[0112] In some embodiments, generating an operation reference use case for a target application includes: presenting data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference use case based on the corresponding first operation information and data information of multiple stages.

[0113] In some embodiments, the data information also indicates at least one of the following: a terminology set referenced for executing the test query (wherein the terminology set includes interpretations of terms used by the user group of the target application), a conversation record between the user and the digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0114] In some embodiments, adjusting the data information includes at least one of the following: updating at least one data set used to execute the test query, or updating at least a portion of at least one data set used to execute the test query.

[0115] In some embodiments, process 700 also includes: in response to receiving a user query for a target application, determining that the running reference case is a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; and executing the user query based on the target running reference case.

[0116] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0117] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0118] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0119] 7B shows a flow chart of a process 710 of information query according to some embodiments of the present disclosure. The process 710 may be implemented at the application management platform 110. The process 710 is described below with reference to FIG.

[0120] In box 711, in response to receiving a user query for a target application, the application management platform 110 determines the running reference case including the test query as the target running reference case of the user query based on the similarity between the user query and the test query exceeding a threshold similarity, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query.

[0121] In block 712 , the application management platform 110 executes the user query based on the target run reference use case.

[0122] In some embodiments, executing a user query based on a target running reference use case includes: providing the target reference use case as part of a prompt word for the user query to a machine learning model; and obtaining an execution result of the user query from the machine learning model.

[0123] In some embodiments, if the user query is from a tester and is implemented in the application testing phase, process 710 also includes: obtaining user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generating additional running reference use cases based on corresponding execution information for executing the user query in multiple phases.

[0124] 7C shows a flow chart of a process 720 of application evaluation according to some embodiments of the present disclosure. The process 720 may be implemented at the application management platform 110. The process 720 is described below with reference to FIG.

[0125] In block 721 , the application management platform 110 generates an evaluation reference case for the target application. The evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result.

[0126] In block 722 , the application management platform 110 selects a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set. The target evaluation sample includes a sample query and a second result generated by the target application for the sample query.

[0127] In block 723 , the application management platform 110 generates an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0128] In some embodiments, generating an evaluation result for the target application includes: providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; obtaining feedback from the machine learning model for the second result in the target evaluation sample; and determining an evaluation result for the target application based on the feedback for the second result.

[0129] In some embodiments, the target application includes at least one function, and the sample query and the user query are both queries for a second function of the at least one function.

[0130] 8A shows a schematic structural block diagram of an apparatus 800 for application debugging according to some embodiments of the present disclosure. Apparatus 800 may be implemented in or included in application management platform 110, for example. Each module / component in apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0131] As shown in the figure, the apparatus 800 includes a test query execution module 801, which is configured to execute a test query for a target application. The apparatus 800 also includes a first operation information presentation module 802, which is configured to present corresponding first operation information of multiple stages of executing the test query, wherein the operation information of the multiple stages at least indicates the output of each of the multiple stages. The apparatus 800 also includes an operation reference case generation module 803, which is configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is used as a reference to determine the result of the user query during the operation of the target application.

[0132] In some embodiments, the reference case generation module 803 includes: a test query re-execution module, configured to determine adjustments to first operating information of at least one of multiple stages; re-execute the test query based on the adjusted first operating information of at least one stage; present corresponding second operating information of multiple stages of the re-executed test query; and generate an operating reference case based on the corresponding second operating information in response to receiving a case generation confirmation.

[0133] In some embodiments, the test query re-execution module includes: a stage execution module, configured to receive a stage running request, the stage running request indicating that the test query be executed from a specified stage among multiple stages, the specified stage being before at least one stage or being included in at least one stage; and re-execute the specified stage among multiple stages and the stages after the specified stage based on the adjusted first running information of at least one stage.

[0134] In some embodiments, the stage execution module includes: a re-execution information presentation module configured to present execution information associated with re-execution of a designated stage and stages subsequent to the designated stage.

[0135] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0136] In some embodiments, the test query re-execution module includes: a first adjustment information receiving module, configured to receive term adjustment information for a first stage of multiple stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and the test query re-execution module is further configured to determine one or more term sets used in the first stage based on the term adjustment information; and re-execute the test query based on the one or more term sets.

[0137] In some embodiments, the test query re-execution module includes: a second adjustment information receiving module, configured to receive use case adjustment information for a second stage among multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and the test query re-execution module is further configured to determine one or more reference use cases to be used in the second stage based on the use case adjustment information; and re-execute the test query based on the one or more reference use cases.

[0138] In some embodiments, the device 800 also includes: a data information presentation module, configured to present data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference use case based on the corresponding first operation information and data information of multiple stages.

[0139] In some embodiments, the data information also indicates at least one of the following: a terminology set referenced for executing the test query, the terminology set including interpretations of terms used by the user group of the target application, a conversation record between the user and the digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0140] In some embodiments, the data information presentation module includes: a data update module configured to update at least one data set used to execute a test query, or to update at least a portion of at least one data set used to execute a test query.

[0141] In some embodiments, the device 800 also includes: a user query execution module, configured to, in response to receiving a user query for a target application, determine that the running reference case is a target running reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity; and execute the user query based on the target running reference case.

[0142] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0143] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0144] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0145] 8B shows a schematic structural block diagram of an apparatus 810 for information query according to some embodiments of the present disclosure. The apparatus 810 may be implemented in or included in the application management platform 110. Each module / component in the apparatus 810 may be implemented by hardware, software, firmware, or any combination thereof.

[0146] As shown, apparatus 810 includes an operation reference case determination module 811, which is configured to, in response to receiving a user query for a target application, determine an operation reference case including the test query as a target operation reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information for multiple stages of the test query. Apparatus 810 also includes a user query execution module 812, which is configured to execute the user query based on the target operation reference case.

[0147] In some embodiments, the user query execution module 812 includes an execution result acquisition module configured to provide the target reference use case as part of the prompt word for the user query to the machine learning model; and obtain the execution result of the user query from the machine learning model.

[0148] In some embodiments, the device 810 also includes an additional operation reference case generation module, which is configured to obtain user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generate additional operation reference cases based on the corresponding execution information for executing the user query in multiple stages.

[0149] 8C shows a schematic structural block diagram of an apparatus 820 for application evaluation according to some embodiments of the present disclosure. Apparatus 820 may be implemented in or included in application management platform 110, for example. Each module / component in apparatus 820 may be implemented by hardware, software, firmware, or any combination thereof.

[0150] As shown in the figure, the device 820 includes an evaluation reference case generation module 821, which is configured to generate an evaluation reference case for the target application, wherein the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result. The device 820 also includes a target evaluation sample selection module 822, which is configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and the queries in the evaluation sample set, wherein the target evaluation sample includes a sample query and a second result generated by the target application for the sample query. The device 820 also includes an evaluation result generation module 823, which is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0151] In some embodiments, the evaluation result generation module 823 includes a model feedback acquisition module, which is configured to provide prompt words generated based on the first result and the evaluation feedback to the machine learning model; obtain feedback on the second result in the target evaluation sample from the machine learning model; and determine the evaluation result for the target application based on the feedback on the second result.

[0152] In some embodiments, the target application includes at least one function, and the sample query and the user query are both queries for a second function of the at least one function.

[0153] FIG9 shows a block diagram of an electronic device 900 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 900 shown in FIG9 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 900 shown in FIG9 may include or be implemented as the application management platform 110 of FIG1 , the device 800 of FIG8A , the device 810 of FIG8B , or the device 820 of FIG8C .

[0154] As shown in FIG9 , electronic device 900 is a general-purpose electronic device. Components of electronic device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 900.

[0155] The electronic device 900 typically includes a plurality of computer storage media. Such media can be any accessible media that the electronic device 900 can access, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 900.

[0156] The electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 9 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 920 may include a computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0157] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 900 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0158] The input device 950 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 960 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 900 may also communicate with one or more external devices (not shown) through the communication unit 940 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 900, or with any device that allows the electronic device 900 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0159] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0160] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0161] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0162] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0163] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0164] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for application debugging, comprising: Execute test queries against the target application; Presenting corresponding first operation information of a plurality of stages of executing the test query, wherein the operation information of the plurality of stages at least indicates outputs of the plurality of stages respectively; as well as Based on the corresponding first operation information of multiple stages of the test query, an operation reference case for the target application is generated, the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

2. The method according to claim 1, wherein generating the operation reference use case comprises: determining an adjustment to first operating information of at least one of the plurality of phases; re-executing the test query based on the adjusted first operation information of the at least one stage; presenting corresponding second run information of re-executing the plurality of stages of the test query; as well as In response to receiving the use case generation confirmation, the operation reference use case is generated based on the corresponding second operation information.

3. The method of claim 2, wherein re-executing the test query comprises: receiving a stage execution request, the stage execution request instructing to execute the test query from a specified stage among the plurality of stages, the specified stage being before or included in the at least one stage; as well as The designated phase and phases subsequent to the designated phase among the plurality of phases are re-executed based on the adjusted first operation information of the at least one phase.

4. The method of claim 3, wherein presenting corresponding second run information of the plurality of stages of re-executing the test query comprises: Run information associated with rerunning of the specified phase and phases subsequent to the specified phase is presented.

5. The method according to claim 1, wherein the operation information of the plurality of stages further indicates at least one of the following: The inputs of the respective stages are The operation time of each of the multiple stages is The operation logs of each of the multiple stages, The operating status of each of the multiple stages, Input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

6. The method of claim 2, wherein determining an adjustment to the first operating information of at least one of the plurality of phases comprises: receiving terminology adjustment information for a first stage of the plurality of stages, the terminology adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including explanations of terms used by a user group of the target application, and Wherein re-executing the test query comprises: Based on the term adjustment information, determining one or more term sets used in the first stage; and The test query is re-executed based on the one or more term sets.

7. The method of claim 2, wherein determining an adjustment to the first operating information of at least one of the plurality of phases comprises: receiving use case adjustment information for a second stage in the plurality of stages, the use case adjustment information indicating adding a reference use case to the second stage or removing a reference use case selected in the second stage, and Wherein re-executing the test query comprises: Determining one or more reference use cases used in the second stage based on the use case adjustment information; and Based on the one or more reference use cases, the test query is re-executed.

8. The method according to claim 1, wherein generating a running reference use case for the target application comprises: Presenting data information while presenting the corresponding first operation information of the plurality of stages, the data information at least indicating at least one data set used to execute the test query; as well as The operation reference use case is generated based on the corresponding first operation information of the multiple stages and the data information.

9. The method according to claim 8, wherein the data information further indicates at least one of the following: a term set referenced for executing the test query, the term set including explanations of terms used by a user group of the target application, A record of a conversation between a user and a digital assistant referenced for executing the test query, A reference prompt word for executing the test query, A condition used to determine candidate results for the test query.

10. The method according to claim 8, wherein the adjustment of the data information comprises at least one of the following: an update to the at least one data set used to execute the test query, or An update to at least a portion of the at least one data set used to execute the test query.

11. The method according to claim 1, further comprising: In response to receiving a user query for the target application, determining the running reference case as a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; as well as A reference use case is run based on the goal to execute the user query. 12 . The method according to claim 11 , wherein the target application includes at least one function, and wherein both the test query and the user query are queries for a first function among the at least one function.

13. The method of claim 1, wherein the plurality of stages comprises: at least one predefined phase, the at least one predefined phase comprising a phase of generating a database query request; as well as At least one phase is determined based on the test query. 14 . The method according to claim 13 , wherein the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

15. A method for information query, comprising: In response to receiving a user query for a target application, based on a similarity between the user query and the test query exceeding a threshold similarity, determining an operation reference case including the test query as a target operation reference case for the user query, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; as well as A reference use case is run based on the goal to execute the user query.

16. The method according to claim 15, wherein executing the user query based on the target running reference use case comprises: Providing the target reference use case to a machine learning model as part of a prompt word for the user query; as well as Obtain execution results of the user query from the machine learning model.

17. A method for application evaluation, comprising: Generate an evaluation reference case for the target application, the evaluation reference case comprising an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; Based on the similarity between the evaluation query and the queries in the evaluation sample set, selecting a target evaluation sample from the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; as well as An evaluation result for the target application is generated based on the first result, the evaluation feedback, and the second result.

18. The method according to claim 17, wherein generating the evaluation result for the target application comprises: Providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; Obtaining feedback for the second result in the target evaluation sample from the machine learning model; as well as Based on the feedback on the second result, an evaluation result for the target application is determined. 19 . The method according to claim 17 , wherein the target application includes at least one function, and wherein both the sample query and the user query are queries for a second function among the at least one function.

20. A device for application debugging, comprising: A test query execution module, configured to execute a test query for a target application; A first operation information presentation module is configured to present corresponding first operation information of a plurality of stages of executing the test query, wherein the operation information of the plurality of stages at least indicates the output of each of the plurality of stages; as well as An operation reference case generation module is configured to generate an operation reference case for the target application based on corresponding first operation information of multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

21. A device for information query, comprising: an operation reference case determination module, configured to, in response to receiving a user query for a target application, determine an operation reference case including the test query as a target operation reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; as well as The user query execution module is configured to execute the user query by running a reference use case based on the target.

22. A device for application evaluation, comprising: An evaluation reference case generation module is configured to generate an evaluation reference case for the target application, wherein the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; A target evaluation sample selection module is configured to select a target evaluation sample from the evaluation sample set based on a similarity between the evaluation query and a query in the evaluation sample set, wherein the target evaluation sample includes a sample query and a second result generated by the target application for the sample query; as well as The evaluation result generating module is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback and the second result.

23. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 19 when executed by the at least one processing unit.

24. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 19.

Images