Application debugging methods, information query methods, and application evaluation methods

Abstract

The embodiments of the present disclosure provide a method for application debugging, a method for information query, a method for application evaluation, an apparatus, a device, and a storage medium. The method for application debugging includes: executing a test query for a target application; presenting corresponding first operation information of multiple stages of executing the test query, the operation information of the multiple stages at least indicating the output of each of the multiple stages; and generating an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, the operation reference case including the test query and the operation information of the multiple stages, and the operation reference case being referenced in the operation of the target application to determine the result of the user query. In this way, the application creator can improve the quality of services provided by the target application without modifying the code, thereby simplifying the debugging process of the target application.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly to a method for application debugging, a method for information querying, a method for application evaluation, an apparatus, a device, and a computer-readable storage medium. Background Art

[0002] With the development of information technology, various terminal devices can provide people with a variety of services in work and life. Applications that provide these services can be deployed on these devices. These applications present content and interact with users through their user interfaces, meeting their needs. In some cases, users may initiate queries within applications. Therefore, improving the quality of query services provided to users is a key concern. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for application debugging is provided. The method includes: executing a test query for a target application; presenting corresponding first operation information for multiple stages of executing the test query, wherein the operation information for the multiple stages at least indicates outputs of each of the multiple stages; and generating an operation reference case for the target application based on the corresponding first operation information for the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information for the multiple stages, and the operation reference case is used as a reference to determine the result of the user query during the operation of the target application.

[0004] In a second aspect of the present disclosure, a method for information query is provided. The method includes: in response to receiving a user query for a target application, determining, based on a similarity between the user query and a test query exceeding a threshold similarity, a running reference case including the test query as a target running reference case for the user query, wherein the running reference case includes the test query and corresponding running information for multiple stages of the test query; and executing the user query based on the target running reference case.

[0005] In a third aspect of the present disclosure, a method for application evaluation is provided. The method includes: generating an evaluation reference case for a target application, the evaluation reference case including an evaluation query, a first result generated by the target application in response to the evaluation query, and evaluation feedback on the first result; selecting a target evaluation sample from the evaluation sample set based on similarity between the evaluation query and queries in an evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application in response to the sample query; and generating an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0006] In a fourth aspect of the present disclosure, a device for application debugging is provided. The device includes: a test query execution module configured to execute a test query for a target application; a first operation information presentation module configured to present corresponding first operation information of multiple stages of executing the test query, wherein the operation information of the multiple stages at least indicates the output of each of the multiple stages; and an operation reference case generation module configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced during the operation of the target application to determine the result of the user query.

[0007] In a fifth aspect of the present disclosure, a device for information query is provided. The device includes: an operation reference case determination module configured to, in response to receiving a user query for a target application, determine an operation reference case including a test query as a target operation reference case for the user query based on a similarity between the user query and a test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; and a user query execution module configured to execute the user query based on the target operation reference case.

[0008] In a sixth aspect of the present disclosure, a device for application evaluation is provided. The device includes: an evaluation reference case generation module configured to generate an evaluation reference case for the target application, the evaluation reference case including an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; a target evaluation sample selection module configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; and an evaluation result generation module configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0009] In a seventh aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the methods of the first and second aspects.

[0010] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect and the method of the second aspect are implemented.

[0011] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0014] Figure 2 An example of a page showing debug entry points for various functions in a target application according to some embodiments of the present disclosure is shown;

[0015] Figure 3 An example of a page showing test query records for a specific function in a target application according to some embodiments of the present disclosure is shown;

[0016] Figure 4 An example of a page for debugging a target application according to some embodiments of the present disclosure is shown;

[0017] Figure 5 An example of a page displaying data information according to some embodiments of the present disclosure is shown;

[0018] Figure 6A An example of a page of a use case library for a specific function in a target application according to some embodiments of the present disclosure is shown;

[0019] Figure 6B An example of a page showing use case details according to some embodiments of the present disclosure is shown;

[0020] Figure 7A A flowchart illustrating a process of application debugging according to some embodiments of the present disclosure is shown;

[0021] Figure 7B A flowchart showing a process of information query according to some embodiments of the present disclosure is shown;

[0022] Figure 7C A flowchart showing a process of application evaluation according to some embodiments of the present disclosure is shown;

[0023] Figure 8A A schematic structural block diagram of an apparatus for application debugging according to some embodiments of the present disclosure is shown;

[0024] Figure 8B A schematic structural block diagram of an apparatus for information query according to some embodiments of the present disclosure is shown;

[0025] Figure 8C A schematic structural block diagram of an apparatus for application evaluation according to some embodiments of the present disclosure is shown; and

[0026] Figure 9 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0028] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below.

[0029] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and authorization should be obtained from relevant users in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to the software or hardware such as the electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message may be sent to the relevant user in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0034] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions. The activation of the relevant functions of the digital assistant in this disclosed embodiment, the data acquired, the processing and storage of the data, etc., shall all be authorized in advance by the user and other rights holders associated with the user, and shall comply with the provisions of relevant laws, regulations and agreement rules between rights holders.

[0036] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0037] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. Environment 100 includes an application management platform 110, which can support the creation and / or execution of applications. In some embodiments, the portion of application management platform 110 that supports application creation can also be referred to as an application creation portion. In some embodiments, the portion of application management platform 110 that supports application execution can also be referred to as an application execution portion.

[0038] like Figure 1 As shown, the application creation part can provide the user 105 with an environment for creating and publishing applications. The user 105 can be referred to as an application creation user or creator. In some embodiments, the application creation part can be a low-code platform that provides a collection of tools for application creation. The application creation part can support visual development of various types of applications, so that developers can skip the manual coding process and speed up the application development cycle and cost. The application creation part can support any appropriate platform for users to develop one or more types of applications, for example, it can include a platform based on Application Platform as a Service (aPaaS). Such a platform can support users to develop applications efficiently and realize operations such as application creation and application function adjustment.

[0039] The application creation part can be deployed locally on the terminal device of user 105, and / or can be supported by a server-side device. For example, the terminal device of user 105 can run a client of the application creation part, which can support the interaction between the user and the application creation part provided by the server. In the case where the application creation part runs locally on the user's terminal device, user 105 can directly use the terminal device to interact with the local application creation part. In the case where the application creation part runs on the server-side device, the server-side device can realize the service provision of the client running in the terminal device based on the communication connection between the terminal device. The application creation part can present a corresponding page 130 to the user 105 based on the operation of the user 105 to output and / or receive information related to application creation to the user 105 and / or from the user 105.

[0040] In some embodiments, the application creation portion can be associated with a corresponding database, which stores the data or information required for the application creation process supported by the application creation portion. For example, the database can store the code and description information corresponding to each functional module used to make up the application. The application creation portion can also perform operations such as calling, adding, deleting, and updating the functional modules in the database. The database can also store operations that can be performed on different functional blocks. For example, in a scenario where an application is to be created, the application creation portion can call the corresponding functional blocks from the database to build the application.

[0041] In an embodiment of the present disclosure, a user 105 can create a target application 120 as needed in the application creation section and publish the target application 120. The target application 120 can be published to any appropriate application execution section, as long as the application execution section can support the operation of the target application 120. After publication, the target application 120 can be used to be operated by one or more users 145. The user 145 can be referred to as the end user of the target application 120. In some embodiments, the target application 120 may include or be implemented as a digital assistant 122.

[0042] The digital assistant 122 can be configured to have an intelligent conversation. Figure 1 In the example shown, the digital assistant 122 can be integrated into the target application 120 as a part of the target application 120 to assist in executing task processing within the target application 120. In other examples, the digital assistant 122 can be configured as an independently running application, such as a web application or other types of applications. In such an example, the digital assistant 122 and the target application 120 can be regarded as the same application. The digital assistant 122 is provided to assist users with various task processing needs in different applications and scenarios. During the interaction with the digital assistant 122, the user inputs an interactive message, and the digital assistant 122 provides a reply message in response to the user input. Generally, the digital assistant 122 can support users to input questions in natural language, and perform tasks and provide replies based on the understanding of natural language input and logical reasoning ability.

[0043] In some embodiments, digital assistant 122 can interact with user 145 as a contact. For example, digital assistant 122 can be implemented in an instant messaging (IM) application. Digital assistant 122 can interact with user 145 in a single chat session with user 145. In some embodiments, digital assistant 122 can interact with multiple users in a group chat session including multiple users.

[0044] For each user 145, the client of the application running portion can present an interaction window 142 of the target application 120 or the digital assistant 122 in the client interface, such as a conversation window with the digital assistant 122. The user 145 can enter a conversation message in the conversation window, and the target application 120 can determine a reply message of the digital assistant 122 based on the created configuration information and present it to the user in the interaction window 142. In some embodiments, depending on the configuration of the target application 120, the interaction message with the target application 120 can include messages in multimodal forms, such as text messages (e.g., natural language text), voice messages, image messages, video messages, etc.

[0045] Similar to the application creation part, the application running part can be deployed locally on the terminal device of each user 145, and / or can be supported by a server-side device. For example, the terminal device of user 145 can run a client with the application running part, which can support the interaction between the user and the application running part provided by the server. In the case where the application running part runs locally on the user's terminal device, the user 145 can directly use the terminal device to interact with the local application running part. In the case where the application running part runs on the server-side device, the server-side device can realize the service provision of the client running in the terminal device based on the communication connection between the terminal device. The application running part can present the corresponding application page to the user 145 based on the operation of the user 145, so as to output and / or receive information related to the use of the application to the user 145 and / or receive information from the user 145 from the user 145.

[0046] In some embodiments, at least some of the functionality of the target application 120 and / or at least some of the functionality of the digital assistant 122 in the target application 120 can be implemented based on a model. During the creation or execution of the target application 120, one or more models 155, such as the capabilities of the models 155, can be invoked. In the target application 120, the digital assistant 122 can utilize the models 155 to understand user input and provide responses to the user based on the output of the models 155.

[0047] During the creation process, the application creation platform 110 tests the target application 120 using the model 155 to ensure that the target application 120's operating results meet expectations. During operation, in response to various user requests for the target application 120, the application execution component may need to use the model 155 to determine the user's response.

[0048] Although shown as being independent of the application management platform 110, one or more models 155 can run on the application management platform 110 or other remote servers. In some embodiments, the model 155 can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model can be based on a language model (LM). A language model can have question-answering capabilities by learning from a large amount of corpus. The model 155 can also be based on other appropriate models.

[0049] The management platform 110 can run on appropriate electronic devices. The electronic devices here can be any type of device with computing capabilities, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can, for example, include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and so on. In some embodiments, the management platform 110 can be implemented based on cloud services.

[0050] It should be understood that the structure and functionality of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Figure 1 A single user interacting with the application creation portion and a single user interacting with the application running portion are shown, but in reality multiple users can access the application management platform 110 to each create a digital assistant, and each digital assistant can be used to interact with multiple users.

[0051] When a user initiates a query request within an app, the app can leverage machine learning models to provide query services. Initially, the quality of the query service provided by the app may be low, requiring multiple rounds of optimization to improve its effectiveness. Traditional optimization requires system improvements at the code level, which is costly, time-consuming, and prone to errors.

[0052] To this end, an application debugging solution is provided according to an embodiment of the present disclosure. According to various embodiments of the present disclosure, a test query is executed for a target application. Corresponding first operation information of multiple stages of executing the test query is presented, and the operation information of the multiple stages at least indicates the output of each of the multiple stages. Based on the corresponding first operation information of the multiple stages of the test query, an operation reference case for the target application is generated. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced during the operation of the target application to determine the result of the user query.

[0053] In various embodiments of the present disclosure, application creators can generate a reference run case for the target application based on the corresponding operational information from multiple stages of executing a test query. This reference run case can then be used as a reference when generating query results for user queries initiated by end users. In this way, application creators can improve the quality of services provided by the target application without modifying the code, simplifying the target application tuning process.

[0054] Some example embodiments of the present disclosure will be described in detail below with reference to the examples in the accompanying drawings. It should be understood that the pages shown in the accompanying drawings are merely examples, and a variety of page designs may exist. The various graphical elements in a page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present. The embodiments of the present disclosure are not limited in this respect.

[0055] The following further describes an exemplary embodiment of the present disclosure with reference to the accompanying drawings. In the following examples, for the sake of discussion, the description is from the perspective of the application management platform, for example Figure 1 The application management platform 110 is shown. The page presented by the application management platform 110 can be presented via the terminal device of the user 105, and the user input can be received via the terminal device of the user 105.

[0056] Figure 2 FIG. 2 shows an example of a page 200 for debugging entries of various functions in a target application 120 according to some embodiments of the present disclosure. Figure 2 As shown, a page 200 with debugging entries for various functions in the target application 120 is presented on the terminal device of user 105. Page 200 provides a functional area 210, which can be used for various functions in the target application. The functions in the target application may include, for example, an application data analysis function, an application data update function, a knowledge question and answer function, etc. However, it should be understood that this is merely an exemplary function and is not intended to be limiting. The target application here can be any suitable application, such as a human resources application, a CRM (Customer Relationship Management) application, a financial application, etc. Taking the debugging of the application data analysis function in the target application as an example, user 105 triggers the debugging control 220 of the data analysis function, for example, user 105 clicks on the debugging control 220. In response to this, a page of test query records can be presented to user 105.

[0057] Figure 3 FIG. 3 shows an example of a page 300 of test query records for a specific function in a target application according to some embodiments of the present disclosure. Figure 3As shown, a page 300 of test query records for a specific function in the target application can be presented on the terminal device of user 105. Page 300 records the test queries previously created by user 105, and user 105 debugs the corresponding function by entering the test query. In some embodiments, user 105 can re-debug the previously created test query. For example, user 105 clicks on control 310 to re-execute the test query "What is the quantity of goods with a unit price lower than XX yuan?" to debug the corresponding function in the target application. In response to this, a debugging page can be presented to user 105. In some embodiments, user 105 can trigger the execution of a new test query. For example, user 105 clicks on the newly added control 320 to enter a new test query to debug the corresponding function in the target application. In response to this, user 105 can enter the debugging page.

[0058] Figure 4 FIG4 shows an example of a page 400 for debugging a target application according to some embodiments of the present disclosure. The application management platform 110 may execute a test query for the target application. Figure 4 For example, user 105 enters a test query statement in text area 410 corresponding to the test query and clicks control 420 to execute a test query for the target application. The test query statement entered by user 105 may be, for example, "What is the quantity of products with a unit price lower than XX yuan?" or "What is the percentage of products produced in XX? Draw a pie chart." The application management platform 110 may present corresponding first execution information for multiple stages of executing the test query in staged debugging area 430. The multiple stage execution information at least indicates the output of each of the multiple stages.

[0059] After executing a test query against a target application, application management platform 110 may present corresponding first execution information for multiple stages of executing the test query in staged debugging area 430. The execution information for each stage at least indicates the output of each stage. For example, staged debugging area 430 displays execution information for five stages, each requiring the execution of a task. After the tasks are completed, corresponding outputs 440, 442, 444, 446, and 448 are generated. For example, output 444 of the third stage may include the data tables required to execute the test query.

[0060] In some embodiments, the operation information of multiple stages may further indicate at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, and the operation status of each of the multiple stages (i.e., whether the operation is completed). In some embodiments, due to the limitations on the input information of the machine learning model (e.g., the length of the input information), the operation information of multiple stages may further indicate the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages. In this way, user 105 can make the input information provided to the machine learning model meet the requirements of the machine learning model, thereby ensuring the stable operation of the machine learning model.

[0061] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include at least one predefined stage, and at least one predefined stage includes a stage for generating a database query request. The at least one predefined stage may be some debugging stages pre-set by the application management platform 110 according to the characteristics and / or corresponding functions of the target application. The purpose of the application management platform 110 performing a test query for the target application may be to generate a better database query request to return more accurate query results to the user. Therefore, the at least one predefined stage may also include a stage for generating a database query request, which may generate an accurate and efficient database query request based on the output of the previous predefined stage. The database query request here may be, for example, an SQL (Structured Query Language) statement.

[0062] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include, in addition to at least one predefined stage, at least one stage determined based on a test query. After the application management platform 110 executes a database query request and obtains the corresponding data, it may also determine at least one stage based on the requirements of the test query. In one example, the test query is "What is the number of products with a unit price less than XX yuan?" This test query does not require additional data analysis, so the application management platform 110 may only determine one stage, such as the data review stage, in which the application management platform 110 may output the number of products with a unit price less than XX yuan. In another example, the test query is "What is the percentage of products produced in XX? Draw a pie chart?" This test query requires additional data analysis, so the application management platform 110 may generate multiple stages, such as a chart generation stage and a data insight stage. The application management platform 110 may output a pie chart showing the percentage of products produced in XX to the chart generation stage. The application management platform 110 may also output data mining and analysis on the percentage of products produced in XX to the data insight stage. In this way, the application management platform 110 can determine different stages after the predefined stage based on the test query, which can meet different query requirements of users and improve the query experience of users.

[0063] In some embodiments, the test query executed by the application management platform 110 for the target application is a query for the first function in the target application. Because different functions in the target application have significant differences, debugging one function in the target application may not be applicable to another function in the target application. Therefore, the test query is a query for a specific function in the target application.

[0064] In some embodiments, the predefined phases may differ for different functions in the target application. For example, taking the first function as an example, at least one predefined phase may be configured for the first function in the target application. The application management platform 110 may configure at least one different predefined phase based on the characteristics of different functions in the target application. For example, for the application data analysis function in the target application, the application management platform 110 may configure five predefined phases: recall and select terms, recall and select reference use cases, recall and select target datasets, recall and select fields belonging to the target dataset, and generate and execute database query requests. The recall and select terms phase may recall and select interpretations of terms used in the test query. The recall and select reference use cases phase may recall and select reference use cases similar to the test query. The recall and select target dataset phase may recall and select the dataset required for executing the test query. The recall and select fields belonging to the target dataset phase may recall and select the fields within the dataset required for executing the test query. The generate and execute database query request phase may generate and execute a database query request based on the outputs of the previous four phases.

[0065] The above describes the phase division using the data analysis function as an example, but this division is merely exemplary. Furthermore, for other functions (e.g., data updates, knowledge questions, etc.), appropriate phase divisions can be made based on the characteristics of the corresponding functions. In this way, different functions in the target application can be debugged separately using test queries, fully considering the differences between different functions and effectively improving the service quality provided by different functions.

[0066] After presenting the corresponding first operation information of multiple stages of executing the test query, the application management platform 110 can generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of multiple stages. The operation reference case is used as a reference to determine the result of the user query during the operation of the target application. For example, the user 105 can first click the tuned control 450 on the page 400 to confirm that the test query and the corresponding operation information have been tuned, and then click the save control 460 to generate the operation reference case. After the operation reference case is generated, the application management platform 110 can refer to the operation reference case to determine the result of the user query when receiving the user query of the user 145.

[0067] In some embodiments, the application management platform 110 may determine an adjustment to the first operation information of at least one of the multiple stages based on the corresponding first operation information of the multiple stages of the test query. Then, the application management platform 110 may re-execute the test query based on the adjusted first operation information of at least one stage. The application management platform 110 may present the corresponding second operation information of the multiple stages of the re-executed test query. If a use case generation confirmation is received, the application management platform 110 may generate a reference use case to be run based on the corresponding second operation information. After the first operation information of at least one stage is adjusted, the application management platform 110 may re-execute the test query and present the corresponding second operation information of the multiple stages of the re-executed test query in the staged debugging area 430. Reference Figure 4 The application management platform 110 may receive a use case generation confirmation through the user 105 triggering the tuned control 450 and the saved control 460 to generate an operation reference use case based on the second operation information.

[0068] In some embodiments, the application management platform 110 may receive terminology adjustment information from the user 105 for the first stage of multiple stages, and re-execute the test query based on the terminology adjustment information. The terminology adjustment information indicates adding a term set to the first stage or removing a term set selected in the first stage. The term set includes explanations of the terms used by the user group of the target application. The terms used by the user group may be used within the scope of the group, but are not common terms, so it is necessary to help the machine learning model understand such terms. The explanation of such a term may include a literal explanation of the term, and may also include a common term corresponding to the term.

[0069] Exemplarily, the first stage of executing the test query may select a term set related to the test query, and the user 105 may add or remove a term set in the first stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods sold through the C2C channel?", where C2C is a specialized term. The term set selected in the first stage may include an explanation of C2C, and the user 105 may add other term sets or delete the term set selected in the first stage. After adjusting the term set, the application management platform 110 may determine one or more term sets used in the first stage based on the term adjustment information, and then re-execute the test query based on the one or more term sets.

[0070] In some embodiments, the application management platform 110 can receive use case adjustment information for the second phase of multiple phases and re-execute the test query based on the use case adjustment information. The use case adjustment information indicates the addition of reference use cases to the second phase or the removal of reference use cases selected in the second phase. For example, the second phase of executing the test query can select use cases similar to the test query, and the user 105 can add or remove reference use cases in the second phase. For example, if the user 105's test query for the target application is "How many products have a unit price less than 5,000 yuan?", the second phase can select reference use cases similar to the test query based on similarity calculation. The reference use case selected in the second phase can be "How many products have a unit price less than 8,000 yuan?" The user 105 can add other reference use cases or move the reference use cases selected in the second phase. After adjusting the reference use cases, the application management platform 110 can determine one or more reference use cases to be used in the second phase based on the use case adjustment information, and then re-execute the test query based on the one or more reference use cases.

[0071] In some embodiments, the application management platform 110 may receive data set adjustment information for the third stage of the multiple stages and re-execute the test query based on the data set adjustment information. For example, the third stage of executing the test query may select the data set required to execute the test query, and the user 105 may add or remove data sets in the third stage. For example, the test query of the user 105 for the target application is "What is the number of products with a unit price of less than 5,000 yuan?" The data set selected in the third stage may be a product analysis table. The user 105 may add other data sets or delete the data set selected in the third stage.

[0072] In some embodiments, the application management platform 110 can also receive data set field adjustment information for the fourth stage among the multiple stages, and re-execute the test query based on the data set field adjustment information. Exemplarily, the fourth stage of executing the test query can select the fields of the data set required to execute the test query, and the user 105 can add fields or remove fields in the fourth stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods with a unit price of less than 5,000 yuan?", and the fields of the data set selected in the fourth stage are the product number field, unit price field, origin field and production date field in the product analysis table. The user 105 can add other fields or delete the fields selected in the fourth stage. In this way, the user can adjust the various stages of the test query through a graphical interface, which facilitates the user's understanding of debugging and simplifies the debugging process of the target application.

[0073] In some embodiments, the application management platform 110 may receive a stage operation request after determining the adjustment to the first operation information of at least one stage. The stage operation request indicates that a test query is to be executed from a specified stage among multiple stages, where the specified stage is before at least one stage or is included in at least one stage, and based on the adjusted first operation information of at least one stage, the specified stage among the multiple stages and the stage after the specified stage are re-executed. Exemplarily, the user 105 adjusts the second stage, and the specified stage may be the first stage before the second stage, or the second stage. In the case where the specified stage is the first stage, the application management platform 110 may execute the first stage and the stage after the first stage (i.e., all stages) based on the adjusted first operation information of at least the second stage.

[0074] For example, reference Figure 4 For each of the five stages, a trigger entry for starting to re-execute the stage is provided, such as trigger controls 421, 422, 423, 424, and 425. The user 105 can trigger control 420 to execute all the stages from the beginning. In the case where the designated stage is the second stage, the application management platform 110 can execute the second stage and the stages after the second stage based on the adjusted first running information of at least the second stage. For example, continue to refer to Figure 4 , the user 105 can trigger the control 422 to execute the second stage and the stage after the second stage. In this way, the user can flexibly run different stages and verify the effectiveness of the adjustments made to different stages by comparing the output results of running different stages. It is understandable that although Figure 4 The corresponding stages are executed by triggering controls 421, 422, 423, 424, and 425, but this triggering method is only illustrative. User 105 can also trigger the corresponding stages in other ways, such as by user input representing natural language, to trigger the execution of the corresponding stages.

[0075] In some embodiments, after re-executing the test query, the application management platform 110 may present operation information associated with the re-execution of the specified stage and the stages subsequent to the specified stage in the staged debugging area 430. For example, if the specified stage is the second stage, operation information associated with the re-execution of the second stage and the stages subsequent to the second stage may be presented in the staged debugging area 430.

[0076] In some embodiments, the application management platform 110 may also present data information while presenting the corresponding first operation information of the multiple stages, the data information at least indicating at least one data set used to execute the test query, and generate an operation reference case based on the corresponding first operation information of the multiple stages and the data information. Figure 4 The application management platform 110 may display data information for executing test queries in the data information display area 470. In addition, the application management platform 110 may generate an operation reference case based on the data information or adjustments to the data information.

[0077] In some embodiments, the adjustment of the data information may include updating the at least one data set used to perform the test query. For example, a data set used to perform the test query, namely, a commodity analysis table, is displayed in the data information display area 470. Figure 4 , user 105 can trigger add control 472 to update the data set.

[0078] In some embodiments, the adjustment of the data information may include updating at least a portion of at least one data set used to perform the test query. Here, the portion of the data set may indicate a field description in the data set. For example, a data set used to perform the test query, namely, a commodity analysis table, is displayed in the data information display area 470. Figure 4 , user 105 can trigger the modification control 474 to update the field description in the data set.

[0079] In some embodiments, the data information further indicates a terminology set referenced for executing the test query, the terminology set including explanations of terms used by the user community of the target application. Figure 5 , Figure 5 An example of a page 500 displaying data information according to some embodiments of the present disclosure is shown. The page 500 includes a term set display area 510, and the user 105 can adjust the content in the term set display area 510.

[0080] In some embodiments, the data information further indicates a conversation record between the user and the digital assistant referenced for executing the test query. Figure 5 Page 500 includes a conversation record display area 520 , and user 105 can adjust the content in the conversation record display area 520 .

[0081] In some embodiments, the data information further indicates a reference prompt word for executing the test query. Figure 5 The page 500 includes a reference prompt word display area 530, and the user 105 can adjust the content in the reference prompt word display area 530. By adjusting the reference prompt word, better guidance can be provided to the machine learning model, helping the machine learning model to better complete the user's task.

[0082] In some embodiments, the data information also indicates conditions for determining candidate results for the test query. Figure 5, the page 500 includes a condition area 540, and the user 105 can adjust the similarity and the number of words to search in the condition area 540. For example, in a knowledge question answering scenario, the digital assistant will search for questions similar to the user's based on similarity, and provide a predetermined number of search results to the user 145 according to the number of single searches.

[0083] refer to Figure 6A , Figure 6A An example of a page 600 for a use case library of a specific function in a target application according to some embodiments of the present disclosure is shown. Page 600 presents a use case library that includes generated reference use cases. User 105 triggers a detail control 602 to jump to the use case details page corresponding to the use case. Figure 6B . Figure 6B An example of a use case details page 610 according to some embodiments of the present disclosure is shown. The page 610 may include a test query and corresponding first run information 614 of multiple stages.

[0084] It can be understood that, according to the embodiments described above, an operation reference use case set including one or more operation reference use cases can be generated.

[0085] In some embodiments, after the target application 120 is published, during the operation of the target application 120, a reference run case can be utilized to provide services to the user 145. For example, in response to receiving a user query for the target application, the application management platform 110 can search for a test query similar to or matching the user query from a collection of reference run cases. Based on the similarity between the user query and a test query exceeding a threshold similarity, the application management platform 110 can determine the reference run case as the target reference run case for the user query and execute the user query based on the target reference run case. After generating the reference run case, the application management platform 110 receives a user query for the target application initiated by the user 145. The application management platform 110 can determine the similarity between the user query and the test query through similarity calculation and, based on the similarity, determine the reference run case as the target reference run case for the user query. After determining the target reference run case, the application management platform 110 can use the target reference run case to execute the user query. In this way, by referencing the reference run case to determine the results of the user query, more accurate query results can be provided to the user, thereby improving user satisfaction.

[0086] In some embodiments, a machine learning model can be used to execute user queries. For example, the application management platform 110 can provide the determined target reference use case to the machine learning model as part of the prompt word for the user query. The application management platform 110 can then obtain the execution result of the user query from the machine learning model. It should be noted that the target reference use case here is selected from the set of running reference use cases generated during the application creation phase or the application testing phase. Such target reference use cases are generated for debugging purposes, rather than from use cases when users actually use the application.

[0087] In some embodiments, the user using target application 120 may be a tester, meaning that the user query may originate from a tester. In this case, if the tester provides positive feedback on a query result, a reference run case can be generated based on the user query and the corresponding query result. For example, operational information from multiple stages of the operational process leading to the query result can be stored as a reference run case. In this way, user feedback is leveraged to enrich the source of reference run cases.

[0088] In some embodiments, the target application includes at least one function, and both the test query and the user query are queries targeting a first function of the at least one function. The target application may include functions such as application data analysis, application data update, and knowledge question answering. The test query and the user query need to target the same function. For example, the test query and the user query may both target the application data analysis function.

[0089] During creation or testing of the target application 120, it may be necessary to evaluate the performance of the target application 120. For example, in the case where the target application includes multiple functions, it may be necessary to evaluate one or some of the functions.

[0090] To this end, in some embodiments, the application management platform 110 may maintain a set of evaluation samples for use in evaluating the target application 120, for example, evaluating one or more functions of the target application 120, such as a data analysis function, a data update function, a knowledge question and answer function, etc. Each evaluation sample in the evaluation sample set may include a sample query and a result (also referred to as a sample query result or a second result) generated by the target application 120 in response to the sample query.

[0091] In some embodiments, to obtain evaluation results for a function in the target application 120, a machine learning model can be used to obtain feedback on sample query results in the evaluation sample. For example, the machine learning model can score the sample query results generated by the target application 120 based on one or more dimensions.

[0092] The machine learning model can give feedback in any suitable manner. As an example, an automatic evaluation method is described. The data source (e.g., knowledge base, data set, etc.) used by the target application 120 to provide services can be provided to the machine learning model, so that the machine learning model extracts queries and corresponding query results therefrom as correct examples. For example, for the knowledge question and answer function, questions and corresponding answers can be extracted. Then, the extracted queries can be executed in the target application 120, and the execution results are generated by the target application 120. Furthermore, the machine learning model can give feedback on the execution results based on the answers and execution results in various suitable dimensions, such as scoring. Examples of evaluation dimensions may include but are not limited to helpfulness to users, language fluency, factual correctness, logic, etc.

[0093] Thus, the machine learning model provides feedback based on the extracted questions and execution results to determine the evaluation results for the target application 120. For example, if the evaluation sample set is for a knowledge question and answer function, the evaluation results for that function can be obtained. In this way, the user 105 can debug the target application 120 based on the evaluation results, for example, debugging one or more functions.

[0094] During the creation or testing of target application 120, the creator or tester may enter some evaluation queries. Target application 120 can generate corresponding query results for the evaluation queries, also known as first results or evaluation query results. The creator or tester can then provide feedback on the evaluation query results, also known as evaluation feedback.

[0095] In some embodiments, evaluation feedback can be used in conjunction with the evaluation sample set to provide more accurate evaluation results for the target application 120. For example, the operation management platform 110 can generate an evaluation reference case for the target application 120 based on the evaluation query provided by the creator or tester. The evaluation reference case includes the evaluation query from the creator or tester, the evaluation query result generated by the target application 120 in response to the evaluation query, and evaluation feedback on the evaluation query result. The evaluation feedback can be positive or negative, and can also indicate the user 145's satisfaction with the query result.

[0096] Generally speaking, for similar queries, it is expected that the evaluation results and evaluation feedback are consistent or matching. In view of this, sample queries similar to the evaluation query can be found from the evaluation sample set, for example, with a similarity exceeding a threshold similarity. The evaluation sample including such similar sample queries is also referred to as the target evaluation sample, which also includes the sample results generated by the target application 120 for the sample query.

[0097] The application management platform 110 can generate evaluation results for the target application 120 based on the evaluation query results, evaluation feedback, and sample query results. For example, the scores of the sample query results in the target evaluation sample can be matched with the evaluation feedback. It is understood that the evaluation reference use cases provide examples of what results satisfy the creator or tester and what results dissatisfy the creator or tester.

[0098] In some embodiments, as described above, a machine learning model can be used to perform an evaluation of the target application 120. Accordingly, a prompt word can be generated based on at least a portion of the evaluation reference case (e.g., evaluation feedback and evaluation query results), and the prompt word can be provided to the machine learning model. The machine learning model can generate feedback on the sample query results in the target evaluation sample. For example, the machine learning model can score the sample query results. Based on the feedback, the evaluation results for the target application 120 can then be determined. It can be understood that the evaluation reference case provides examples of what kind of results the creator or tester is satisfied with or what kind of results the creator or tester is dissatisfied with. In this way, the machine learning model can refer to such examples when generating feedback (e.g., scoring) for the sample results. Note that the machine learning model used in the evaluation can be the same as or different from the machine learning model used in the operation of the target application 120.

[0099] In some embodiments, if the target application 120 includes at least one function, the sample queries used for evaluation and the evaluation queries may be queries targeting the same function. For example, the evaluation sample set may be dedicated to evaluating the knowledge question and answer function of the target application 120. In this case, the collected user queries are also targeted at the knowledge question and answer function.

[0100] Figure 7A FIG. 7 is a flow chart illustrating a process 700 for application debugging according to some embodiments of the present disclosure. The process 700 may be implemented at the application management platform 110. The process 700 is described below with reference to FIG.

[0101] In block 701 , the application management platform 110 executes a test query for a target application.

[0102] In block 702 , the application management platform 110 presents corresponding first running information of multiple stages of executing the test query, where the running information of the multiple stages at least indicates outputs of the respective stages.

[0103] In box 703, the application management platform 110 generates an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0104] In some embodiments, generating an operation reference use case includes: determining an adjustment to first operation information of at least one stage among multiple stages; re-executing a test query based on the adjusted first operation information of at least one stage; presenting corresponding second operation information of multiple stages of the re-executed test query; and generating an operation reference use case based on the corresponding second operation information in response to receiving a use case generation confirmation.

[0105] In some embodiments, re-executing the test query includes: receiving a stage running request, the stage running request instructing to execute the test query from a specified stage among multiple stages, the specified stage being before at least one stage or being included in at least one stage; and re-executing the specified stage among the multiple stages and the stages after the specified stage based on the adjusted first running information of at least one stage.

[0106] In some embodiments, presenting corresponding second execution information of the plurality of stages of re-executing the test query includes presenting execution information associated with re-execution of a specified stage and stages subsequent to the specified stage.

[0107] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0108] In some embodiments, determining an adjustment to first run information of at least one of a plurality of stages includes: receiving term adjustment information for a first stage of a plurality of stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and re-executing the test query includes: determining one or more term sets used in the first stage based on the term adjustment information; and re-executing the test query based on the one or more term sets.

[0109] In some embodiments, determining an adjustment to first operating information of at least one stage among multiple stages includes: receiving use case adjustment information for a second stage among the multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and re-executing the test query includes: determining one or more reference use cases used in the second stage based on the use case adjustment information; and re-executing the test query based on the one or more reference use cases.

[0110] In some embodiments, generating an operation reference use case for a target application includes: presenting data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference use case based on the corresponding first operation information and data information of multiple stages.

[0111] In some embodiments, the data information also indicates at least one of the following: a terminology set referenced for executing the test query (wherein the terminology set includes interpretations of terms used by the user group of the target application), a conversation record between the user and the digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0112] In some embodiments, adjusting the data information includes at least one of the following: updating at least one data set used to execute the test query, or updating at least a portion of at least one data set used to execute the test query.

[0113] In some embodiments, process 700 also includes: in response to receiving a user query for a target application, determining that the running reference case is a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; and executing the user query based on the target running reference case.

[0114] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0115] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0116] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0117] Figure 7BFIG. 7 is a flow chart illustrating a process 710 of information query according to some embodiments of the present disclosure. The process 710 may be implemented at the application management platform 110. The process 710 is described below with reference to FIG.

[0118] In box 711, in response to receiving a user query for a target application, the application management platform 110 determines the running reference case including the test query as the target running reference case of the user query based on the similarity between the user query and the test query exceeding a threshold similarity, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query.

[0119] In block 712 , the application management platform 110 executes the user query based on the target run reference use case.

[0120] In some embodiments, executing a user query based on a target running reference use case includes: providing the target reference use case as part of a prompt word for the user query to a machine learning model; and obtaining an execution result of the user query from the machine learning model.

[0121] In some embodiments, if the user query is from a tester and is implemented in the application testing phase, process 710 also includes: obtaining user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generating additional running reference use cases based on corresponding execution information for executing the user query in multiple phases.

[0122] Figure 7C FIG. 7 is a flow chart illustrating a process 720 of application evaluation according to some embodiments of the present disclosure. The process 720 may be implemented at the application management platform 110. The process 720 is described below with reference to FIG.

[0123] In block 721 , the application management platform 110 generates an evaluation reference case for the target application. The evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result.

[0124] In block 722 , the application management platform 110 selects a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set. The target evaluation sample includes a sample query and a second result generated by the target application for the sample query.

[0125] In block 723 , the application management platform 110 generates an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0126] In some embodiments, generating an evaluation result for the target application includes: providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; obtaining feedback from the machine learning model for the second result in the target evaluation sample; and determining an evaluation result for the target application based on the feedback for the second result.

[0127] In some embodiments, the target application includes at least one function, and the sample query and the user query are both queries for a second function of the at least one function.

[0128] Figure 8A 1 shows a schematic structural block diagram of an apparatus 800 for application debugging according to some embodiments of the present disclosure. The apparatus 800 may be implemented in or included in the application management platform 110. Each module / component in the apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0129] As shown in the figure, the apparatus 800 includes a test query execution module 801, which is configured to execute a test query for a target application. The apparatus 800 also includes a first operation information presentation module 802, which is configured to present corresponding first operation information of multiple stages of executing the test query, wherein the operation information of the multiple stages at least indicates the output of each of the multiple stages. The apparatus 800 also includes an operation reference case generation module 803, which is configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is used as a reference to determine the result of the user query during the operation of the target application.

[0130] In some embodiments, the reference case generation module 803 includes: a test query re-execution module, configured to determine adjustments to first operating information of at least one of multiple stages; re-execute the test query based on the adjusted first operating information of at least one stage; present corresponding second operating information of multiple stages of the re-executed test query; and generate an operating reference case based on the corresponding second operating information in response to receiving a case generation confirmation.

[0131] In some embodiments, the test query re-execution module includes: a stage execution module, configured to receive a stage running request, the stage running request indicating that the test query be executed from a specified stage among multiple stages, the specified stage being before at least one stage or being included in at least one stage; and re-execute the specified stage among multiple stages and the stages after the specified stage based on the adjusted first running information of at least one stage.

[0132] In some embodiments, the stage execution module includes: a re-execution information presentation module configured to present execution information associated with re-execution of a designated stage and stages subsequent to the designated stage.

[0133] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0134] In some embodiments, the test query re-execution module includes: a first adjustment information receiving module, configured to receive term adjustment information for a first stage of multiple stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and the test query re-execution module is further configured to determine one or more term sets used in the first stage based on the term adjustment information; and re-execute the test query based on the one or more term sets.

[0135] In some embodiments, the test query re-execution module includes: a second adjustment information receiving module, configured to receive use case adjustment information for a second stage among multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and the test query re-execution module is further configured to determine one or more reference use cases to be used in the second stage based on the use case adjustment information; and re-execute the test query based on the one or more reference use cases.

[0136] In some embodiments, the device 800 also includes: a data information presentation module, configured to present data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference use case based on the corresponding first operation information and data information of multiple stages.

[0137] In some embodiments, the data information also indicates at least one of the following: a terminology set referenced for executing the test query, the terminology set including interpretations of terms used by the user group of the target application, a conversation record between the user and the digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0138] In some embodiments, the data information presentation module includes: a data update module configured to update at least one data set used to execute a test query, or to update at least a portion of at least one data set used to execute a test query.

[0139] In some embodiments, the device 800 also includes: a user query execution module, configured to, in response to receiving a user query for a target application, determine that the running reference case is a target running reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity; and execute the user query based on the target running reference case.

[0140] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0141] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0142] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0143] Figure 8B 1 shows a schematic structural block diagram of an apparatus 810 for information query according to some embodiments of the present disclosure. The apparatus 810 may be implemented in or included in the application management platform 110. Each module / component in the apparatus 810 may be implemented by hardware, software, firmware, or any combination thereof.

[0144] As shown, apparatus 810 includes an operation reference case determination module 811, which is configured to, in response to receiving a user query for a target application, determine an operation reference case including the test query as a target operation reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information for multiple stages of the test query. Apparatus 810 also includes a user query execution module 812, which is configured to execute the user query based on the target operation reference case.

[0145] In some embodiments, the user query execution module 812 includes an execution result acquisition module configured to provide the target reference use case as part of the prompt word for the user query to the machine learning model; and obtain the execution result of the user query from the machine learning model.

[0146] In some embodiments, the device 810 also includes an additional operation reference case generation module, which is configured to obtain user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generate additional operation reference cases based on the corresponding execution information for executing the user query in multiple stages.

[0147] Figure 8C 1 shows a schematic structural block diagram of an apparatus 820 for application evaluation according to some embodiments of the present disclosure. The apparatus 820 may be implemented in or included in the application management platform 110. Each module / component in the apparatus 820 may be implemented by hardware, software, firmware, or any combination thereof.

[0148] As shown in the figure, the device 820 includes an evaluation reference case generation module 821, which is configured to generate an evaluation reference case for the target application, wherein the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result. The device 820 also includes a target evaluation sample selection module 822, which is configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and the queries in the evaluation sample set, wherein the target evaluation sample includes a sample query and a second result generated by the target application for the sample query. The device 820 also includes an evaluation result generation module 823, which is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0149] In some embodiments, the evaluation result generation module 823 includes a model feedback acquisition module, which is configured to provide prompt words generated based on the first result and the evaluation feedback to the machine learning model; obtain feedback on the second result in the target evaluation sample from the machine learning model; and determine the evaluation result for the target application based on the feedback on the second result.

[0150] In some embodiments, the target application includes at least one function, and the sample query and the user query are both queries for a second function of the at least one function.

[0151] Figure 9 1 shows a block diagram of an electronic device 900 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 9 The illustrated electronic device 900 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 9 The electronic device 900 shown may include or be implemented as Figure 1 Application management platform 110, Figure 8A Device 800, Figure 8B device 810, or Figure 8C device 820.

[0152] like Figure 9 As shown, electronic device 900 is in the form of a general electronic device. Components of electronic device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 900.

[0153] The electronic device 900 typically includes a plurality of computer storage media. Such media can be any accessible media that the electronic device 900 can access, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 900.

[0154] The electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 9 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0155] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 900 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0156] The input device 950 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 960 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 900 may also communicate with one or more external devices (not shown) through the communication unit 940 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 900, or with any device that allows the electronic device 900 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0157] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0158] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0159] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0160] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0161] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0162] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for application debugging, comprising: Execute test queries against the target application; Presenting corresponding first execution information of a plurality of stages of executing the test query, wherein the execution information of the plurality of stages at least indicates outputs of each of the plurality of stages and input information provided to the machine learning model when the machine learning model is called in at least one of the plurality of stages; as well as Based on the corresponding first operation information of multiple stages of the test query, an operation reference case for the target application is generated, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

2. The method according to claim 1, wherein generating the operation reference use case comprises: determining an adjustment to first operational information of at least one of the plurality of phases; re-executing the test query based on the adjusted first running information of the at least one stage; presenting corresponding second execution information of the plurality of stages of re-executing the test query; as well as In response to receiving the use case generation confirmation, the operation reference use case is generated based on the corresponding second operation information.

3. The method of claim 2, wherein re-executing the test query comprises: receiving a stage execution request, the stage execution request instructing to execute the test query from a specified stage among the plurality of stages, the specified stage being before or included in the at least one stage; as well as The designated phase and phases subsequent to the designated phase among the plurality of phases are re-executed based on the adjusted first operation information of the at least one phase.

4. The method of claim 3 , wherein presenting corresponding second run information of the plurality of stages of re-executing the test query comprises: Run information associated with reruns of the designated phase and phases subsequent to the designated phase is presented.

5. The method according to claim 1, wherein the operation information of the plurality of stages further indicates at least one of the following: The inputs of the respective stages, The execution time of each of the multiple stages is The operation logs of each of the multiple stages, The operating status of each of the multiple stages.

6. The method of claim 2, wherein determining an adjustment to first operating information of at least one of the plurality of phases comprises: receiving terminology adjustment information for a first stage of the plurality of stages, the terminology adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including explanations of terms used by a user group of the target application, and Re-executing the test query includes: determining one or more terminology sets to be used in the first stage based on the terminology adjustment information; and The test query is re-executed based on the one or more term sets.

7. The method of claim 2, wherein determining an adjustment to first operating information of at least one of the plurality of phases comprises: receiving use case adjustment information for a second phase of the plurality of phases, the use case adjustment information indicating adding a reference use case to the second phase or removing a reference use case selected in the second phase, and Re-executing the test query includes: determining one or more reference use cases to be used in the second stage based on the use case adjustment information; and The test query is re-executed based on the one or more reference use cases.

8. The method according to claim 1, wherein generating an operation reference use case for the target application comprises: Presenting data information while presenting the corresponding first running information of the plurality of stages, the data information at least indicating at least one data set used to execute the test query; as well as The operation reference use case is generated based on the corresponding first operation information of the multiple stages and the data information.

9. The method according to claim 8, wherein the data information further indicates at least one of the following: A term set referenced for executing the test query, wherein the term set includes explanations of terms used by the user community of the target application, A record of the conversation between the user and the digital assistant used to perform the test query, Reference prompt words for executing the test query, Conditions used to determine candidate results for the test query.

10. The method according to claim 8, wherein adjusting the data information comprises at least one of the following: an update to the at least one data set used to execute the test query, or An update to at least a portion of the at least one data set used to execute the test query.

11. The method according to claim 1 , further comprising: In response to receiving a user query for the target application, determining the running reference case as a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; as well as A reference use case is run based on the goal to execute the user query. 12 . The method according to claim 11 , wherein the target application includes at least one function, and wherein both the test query and the user query are queries for a first function among the at least one function.

13. The method of claim 1 , wherein the plurality of stages comprises: at least one predefined phase, the at least one predefined phase comprising a phase of generating a database query request; as well as At least one phase is determined based on the test query. 14 . The method according to claim 13 , wherein the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

15. A method for information query, comprising: In response to receiving a user query for a target application, based on a similarity between the user query and a test query used to debug the target application exceeding a threshold similarity, determining a running reference case including the test query as a target running reference case for the user query, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query, the running information of the multiple stages at least indicating outputs of each of the multiple stages and input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages; as well as A reference use case is run based on the goal to execute the user query.

16. The method according to claim 15, wherein executing the user query by running a reference use case based on the goal comprises: Providing the target operation reference case to a machine learning model as part of a prompt word for the user query; as well as Obtaining execution results of the user query from the machine learning model.

17. A method for application evaluation, comprising: Generating an evaluation reference case for a target application, the evaluation reference case comprising an evaluation query, a first result generated by the target application in response to the evaluation query, and evaluation feedback on the first result, the evaluation feedback indicating user satisfaction with the first result; Selecting a target evaluation sample from the evaluation sample set based on a similarity between the evaluation query and queries in the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; as well as An evaluation result for the target application is generated based on the first result, the evaluation feedback, and the second result.

18. The method according to claim 17, wherein generating the evaluation result for the target application comprises: Providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; Obtaining feedback on the second result in the target evaluation sample from the machine learning model; as well as Based on the feedback on the second result, an evaluation result for the target application is determined. 19 . The method according to claim 17 , wherein the target application includes at least one function, and wherein both the sample query and the user query are queries for a second function among the at least one function.

20. A device for application debugging, comprising: a test query execution module, configured to execute a test query for a target application; a first operation information presenting module configured to present first operation information corresponding to a plurality of stages of executing the test query, wherein the operation information of the plurality of stages at least indicates outputs of the plurality of stages and input information provided to the machine learning model when the machine learning model is called in at least one of the plurality of stages; as well as An operation reference case generation module is configured to generate an operation reference case for the target application based on the corresponding first operation information of multiple stages of the test query. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced in the operation of the target application to determine the result of the user query.

21. A device for information query, comprising: a running reference case determining module configured to, in response to receiving a user query for a target application, determine, based on a similarity between the user query and a test query used to debug the target application exceeding a threshold similarity, a running reference case including the test query as a target running reference case for the user query, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query, the running information of the multiple stages at least indicating outputs of the multiple stages and input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages; as well as The user query execution module is configured to run a reference use case based on the target to execute the user query.

22. A device for application evaluation, comprising: an evaluation reference case generation module configured to generate an evaluation reference case for a target application, the evaluation reference case comprising an evaluation query, a first result generated by the target application in response to the evaluation query, and evaluation feedback on the first result, the evaluation feedback indicating a user satisfaction level with respect to the first result; a target evaluation sample selection module configured to select a target evaluation sample from the evaluation sample set based on a similarity between the evaluation query and queries in the evaluation sample set, the target evaluation sample comprising a sample query and a second result generated by the target application for the sample query; as well as The evaluation result generating module is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback and the second result.

23. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 19 when executed by the at least one processing unit.

24. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method according to any one of claims 1 to 19.

Images