Application debugging method, information query method and application evaluation method

By executing test queries on terminal devices and generating reference test cases, the high cost and time consumption of optimizing query service quality for terminal devices in existing technologies are resolved, achieving the goal of simplifying optimization and improving query service effectiveness.

CN119003326AActive Publication Date: 2024-11-22BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202311570383.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-11-22
Publication Date
2024-11-22
Estimated Expiration
2043-11-22

AI Technical Summary

Technical Problem

Existing technologies for optimizing query service quality on terminal devices suffer from high costs, long processing times, and a high susceptibility to errors.

Method used

By executing test queries against the target application, runtime information at multiple stages is presented, and runtime reference test cases are generated based on this information to determine the results of user queries, simplifying the optimization process without modifying the code.

Benefits of technology

It improved the quality of services provided by the application, simplified the optimization process, and enhanced the effectiveness of the query service.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003326A_ABST
    Figure CN119003326A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an application debugging method, an information query method, an application evaluation method and device, equipment and a storage medium. The application debugging method comprises the following steps: executing a test query for a target application; presenting respective first run information for a plurality of stages in which the test query is performed, the run information for the plurality of stages indicating at least respective outputs of the plurality of stages; based on the corresponding first operation information of the multiple stages of the test query, an operation reference case for the target application is generated, the operation reference case comprises the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to be used for determining the result of the user query. In this way, the application creation user can improve the quality of the service provided by the target application without modifying the code, and the debugging process of the target application is simplified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the computer field, and in particular, to a method for application debugging, a method for information query, a method for application evaluation, an apparatus, a device, and a computer-readable storage medium. Background Art

[0002] With the development of information technology, various terminal devices can provide various services to people in work and life. Applications that provide services can be deployed in terminal devices. Terminal devices present corresponding content and interact with users through the user interface of the application to meet various needs of users. In some cases, users may initiate query requests within the application. Therefore, how to improve the quality of query services provided to users is a problem of concern. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for application debugging is provided. The method includes: executing a test query for a target application; presenting corresponding first operation information of multiple stages of executing the test query, wherein the operation information of the multiple stages at least indicates the output of each of the multiple stages; and generating an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0004] In a second aspect of the present disclosure, a method for information query is provided. The method includes: in response to receiving a user query for a target application, based on the similarity between the user query and the test query exceeding a threshold similarity, determining an operation reference case including the test query as a target operation reference case for the user query, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; and executing the user query based on the target operation reference case.

[0005] In a third aspect of the present disclosure, a method for application evaluation is provided. The method includes: generating an evaluation reference case for the target application, the evaluation reference case including an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; selecting a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; and generating an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0006] In a fourth aspect of the present disclosure, a device for application debugging is provided. The device includes: a test query execution module, configured to execute a test query for a target application; a first operation information presentation module, configured to present corresponding first operation information of multiple stages of executing the test query, the operation information of the multiple stages at least indicating the output of each of the multiple stages; and an operation reference case generation module, configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, the operation reference case including the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0007] In a fifth aspect of the present disclosure, a device for information query is provided. The device includes: an operation reference case determination module, configured to, in response to receiving a user query for a target application, determine an operation reference case including the test query as a target operation reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; and a user query execution module, configured to execute the user query based on the target operation reference case.

[0008] In a sixth aspect of the present disclosure, a device for application evaluation is provided. The device includes: an evaluation reference case generation module, configured to generate an evaluation reference case for the target application, the evaluation reference case including an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; a target evaluation sample selection module, configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and the queries in the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; and an evaluation result generation module, configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0009] In a seventh aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the methods of the first aspect and the second aspect.

[0010] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect and the method of the second aspect are implemented.

[0011] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0014] Figure 2 An example of a page of a debugging entry for each function in a target application according to some embodiments of the present disclosure is shown;

[0015] Figure 3 An example of a page of test query records for a specific function in a target application according to some embodiments of the present disclosure is shown;

[0016] Figure 4 An example of a page for debugging a target application according to some embodiments of the present disclosure is shown;

[0017] Figure 5 An example of a page displaying data information according to some embodiments of the present disclosure is shown;

[0018] Fig. 6A An example of a page of a use case library for a specific function in a target application according to some embodiments of the present disclosure is shown;

[0019] Figure 6B An example of a page showing use case details according to some embodiments of the present disclosure is shown;

[0020] Fig. 7A A flowchart showing a process of application debugging according to some embodiments of the present disclosure is shown;

[0021] Figure 7B A flowchart showing a process of information query according to some embodiments of the present disclosure;

[0022] Figure 7C A flowchart showing a process of application evaluation according to some embodiments of the present disclosure is shown;

[0023] Fig. 8A A schematic structural block diagram of an apparatus for application debugging according to some embodiments of the present disclosure is shown;

[0024] Figure 8B A schematic structural block diagram of an apparatus for information query according to some embodiments of the present disclosure is shown;

[0025] Figure 8C A schematic structural block diagram of an apparatus for application evaluation according to some embodiments of the present disclosure is shown; and

[0026] Fig. 9 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0029] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scopes of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and their authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0032] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to software or hardware such as an electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message is sent to the relevant user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0034] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions. The activation of the digital assistant related functions, the acquired data, the processing and storage methods of the data, etc. in the disclosed embodiment shall obtain the advance authorization of the user and other rights holders associated with the user, and shall comply with the relevant laws, regulations and the agreement rules between the rights holders.

[0036] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0037] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. Environment 100 involves an application management platform 110, which can support the creation of applications and / or the running of applications. In some embodiments, the portion of the application management platform 110 that supports the creation of applications can also be referred to as an application creation portion. In some embodiments, the portion of the application management platform 110 that supports the running of applications can also be referred to as an application running portion.

[0038] like Figure 1 As shown, the application creation part can provide the user 105 with an environment for creating and publishing applications. User 105 can be referred to as an application creation user or creator. In some embodiments, the application creation part can be a low-code platform that provides a collection of tools for application creation. The application creation part can support visual development of various types of applications, so that developers can skip the manual coding process and speed up the application development cycle and cost. The application creation part can support any appropriate platform for users to develop one or more types of applications, for example, it can include a platform based on Application Platform as a Service (aPaaS). Such a platform can support users to efficiently develop applications and implement operations such as application creation and application function adjustment.

[0039] The application creation part can be deployed locally on the terminal device of user 105, and / or can be supported by a server-side device. For example, the terminal device of user 105 can run a client of the application creation part, which can support the interaction between the user and the application creation part provided by the server. In the case where the application creation part runs locally on the user's terminal device, the user 105 can directly interact with the local application creation part using the terminal device. In the case where the application creation part runs on the server-side device, the server-side device can, based on the communication connection with the terminal device, implement service provision for the client running in the terminal device. The application creation part can present a corresponding page 130 to the user 105 based on the operation of the user 105, so as to output to the user 105 and / or receive from the user 105 information related to application creation.

[0040] In some embodiments, the application creation part can be associated with a corresponding database, which stores the data or information required for the application creation process supported by the application creation part. For example, the database can store the code and description information corresponding to each functional module that constitutes the application. The application creation part can also perform operations such as calling, adding, deleting, and updating the functional modules in the database. The database can also store operations that can be performed on different functional blocks. Exemplarily, in a scenario where an application is to be created, the application creation part can call the corresponding functional blocks from the database to build the application.

[0041] In an embodiment of the present disclosure, a user 105 can create a target application 120 as needed on the application creation part and publish the target application 120. The target application 120 can be published to any appropriate application running part, as long as the application running part can support the operation of the target application 120. After publishing, the target application 120 can be used to be operated by one or more users 145. The user 145 can be referred to as the end user of the target application 120. In some embodiments, the target application 120 may include or be implemented as a digital assistant 122.

[0042] The digital assistant 122 may be configured to have an intelligent conversation. Figure 1 In the example shown, the digital assistant 122 can be integrated into the target application 120 as a part of the target application 120 to assist in the task processing within the target application 120. In other examples, the digital assistant 122 can be configured as an independently running application, such as a web application or other types of applications. In such an example, the digital assistant 122 and the target application 120 can be regarded as the same application. The digital assistant 122 is provided to assist users with various task processing requirements in different applications and scenarios. During the interaction with the digital assistant 122, the user inputs an interactive message, and the digital assistant 122 provides a reply message in response to the user input. Generally, the digital assistant 122 can support users to input questions in natural language, and perform tasks and provide replies based on the understanding of natural language input and logical reasoning ability.

[0043] In some embodiments, digital assistant 122 can interact with user 145 as a contact. For example, digital assistant 122 can be implemented in an instant messaging (IM) application. Digital assistant 122 can interact with user 145 in a single chat session with user 145. In some embodiments, digital assistant 122 can interact with multiple users in a group chat session including multiple users.

[0044] For each user 145, the client of the application running part can present an interactive window 142 of the target application 120 or the digital assistant 122 in the client interface, such as a conversation window with the digital assistant 122. The user 145 can enter a conversation message in the conversation window, and the target application 120 can determine the reply message of the digital assistant 122 based on the created configuration information and present it to the user in the interactive window 142. In some embodiments, depending on the configuration of the target application 120, the interactive message with the target application 120 may include a message in a multimodal form, such as a text message (e.g., a natural language text), a voice message, an image message, a video message, and the like.

[0045] Similar to the application creation part, the application running part can be deployed locally on the terminal device of each user 145, and / or can be supported by a server-side device. For example, the terminal device of user 145 can run a client with an application running part, which can support the interaction between the user and the application running part provided by the server. In the case where the application running part runs locally on the user's terminal device, the user 145 can directly interact with the local application running part using the terminal device. In the case where the application running part runs on the server-side device, the server-side device can realize the service supply to the client running in the terminal device based on the communication connection with the terminal device. The application running part can present the corresponding application page to the user 145 based on the operation of the user 145, so as to output and / or receive information related to the use of the application to the user 145.

[0046] In some embodiments, the implementation of at least part of the functions of the target application 120 and / or the implementation of at least part of the functions of the digital assistant 122 in the target application 120 can be implemented based on the model. During the creation or operation of the target application 120, one or more models 155, such as the capabilities of the model 155, can be called. In the target application 120, the digital assistant 122 can use the model 155 to understand the user input and provide a reply to the user based on the output of the model 155.

[0047] During the creation process, the application creation platform 110 needs to use the model 155 to test the target application 120 to determine whether the operation result of the target application 120 meets expectations. During the operation process, in response to different operation requests of the user of the target application 120, the application operation part may need to use the model 155 to determine the response result to the user.

[0048] Although shown as being independent of the application management platform 110, one or more models 155 may run on the application management platform 110, or other remote servers. In some embodiments, the model 155 may be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model may be based on a language model (LM). The language model can have question-answering capabilities by learning from a large amount of corpus. The model 155 may also be based on other appropriate models.

[0049] The management platform 110 can run on appropriate electronic devices. The electronic devices here can be any type of device with computing capabilities, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like. In some embodiments, the management platform 110 can be implemented based on cloud services.

[0050] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Figure 1 A single user interacting with the application creation portion and a single user interacting with the application running portion are shown, but in reality multiple users can access the application management platform 110 to each create a digital assistant, and each digital assistant can be used to interact with multiple users.

[0051] When a user initiates a query request within the application, the application can use the machine learning model to provide query services. At the beginning, the quality of the query service provided by the application may not be high, and multiple rounds of optimization are required to improve the effect of the query service. Traditional optimization requires improving the system at the code level, which is costly, time-consuming, and prone to errors.

[0052] To this end, according to an embodiment of the present disclosure, a solution for application debugging is provided. According to various embodiments of the present disclosure, a test query for a target application is executed. Corresponding first operation information of multiple stages of executing the test query is presented, and the operation information of the multiple stages at least indicates the output of each of the multiple stages. Based on the corresponding first operation information of the multiple stages of the test query, an operation reference case for the target application is generated, the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0053] In various embodiments of the present disclosure, the application creation user only needs to generate an operation reference case for the target application based on the operation information corresponding to the multiple stages of executing the test query. When generating query results for the user query initiated by the terminal user, the operation reference case can be referred to. In this way, the application creation user can improve the quality of service provided by the target application without modifying the code, simplifying the tuning process of the target application.

[0054] Some example embodiments of the present disclosure will be described in detail below with reference to the examples of the accompanying drawings. It should be understood that the pages shown in the accompanying drawings are only examples, and various page designs may actually exist. The various graphical elements in the page may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also exist. The embodiments of the present disclosure are not limited in this respect.

[0055] The following further describes an exemplary embodiment of the present disclosure with reference to the accompanying drawings. In the following examples, for the sake of discussion, the description is made from the perspective of an application management platform, for example Figure 1 The application management platform 110 is shown. The page presented by the application management platform 110 can be presented via the terminal device of the user 105, and the user input can be received via the terminal device of the user 105.

[0056] Figure 2 FIG. 2 shows an example of a page 200 of debugging entries for various functions in a target application 120 according to some embodiments of the present disclosure. Figure 2 As shown, a page 200 with debugging entries for various functions in the target application 120 is presented on the terminal device of user 105. Page 200 provides a functional area 210, which can be used for various functions in the target application. The functions in the target application may include, for example, an application data analysis function, an application data update function, a knowledge question and answer function, etc. However, it should be understood that this is only an exemplary function and is not intended to be limiting. The target application here can be any suitable application, such as a human resources application, a CRM (Customer Relationship Management) application, a financial application, etc. Taking the debugging of the application data analysis function in the target application as an example, the user 105 triggers the debugging control 220 of the data analysis function, for example, the user 105 clicks the debugging control 220. In response to this, a page of test query records can be presented to the user 105.

[0057] Figure 3 An example of a page 300 of test query records for a specific function in a target application according to some embodiments of the present disclosure is shown. Figure 3As shown, a page 300 of test query records for a specific function in a target application can be presented on the terminal device of user 105. Page 300 records the test query previously created by user 105, and user 105 debugs the corresponding function by entering the test query. In some embodiments, user 105 can re-debug a previously created test query. For example, user 105 clicks on control 310 to re-execute the test query "What is the number of goods with a unit price lower than XX yuan?" to debug the corresponding function in the target application. In response to this, a debugging page can be presented to user 105. In some embodiments, user 105 can trigger the execution of a new test query. For example, user 105 clicks on the newly added control 320 to enter a new test query to debug the corresponding function in the target application. In response to this, user 105 can enter the debugging page.

[0058] Figure 4 An example of a page 400 for debugging a target application according to some embodiments of the present disclosure is shown. The application management platform 110 may execute a test query for the target application. Figure 4 For example, the user 105 enters a test query statement in the text area 410 corresponding to the test query and clicks the control 420 to execute a test query for the target application. The test query statement entered by the user 105 may be, for example, "What is the number of goods with a unit price lower than XX yuan?", "What is the proportion of goods with the origin of XX? Draw a pie chart". The application management platform 110 may present the corresponding first operation information of multiple stages of executing the test query in the staged debugging area 430, and the operation information of the multiple stages at least indicates the output of each of the multiple stages.

[0059] After executing the test query for the target application, the application management platform 110 may present corresponding first operation information of multiple stages of executing the test query in the staged debugging area 430, and the operation information of the multiple stages at least indicates the output of each of the multiple stages. For example, the operation information of five stages is shown in the staged debugging area 430, and the five stages need to execute respective tasks. After the task execution is completed, the corresponding outputs 440, 442, 444, 446 and 448 are generated. For example, the output 444 of the third stage may include the data table required to execute the test query.

[0060] In some embodiments, the operation information of multiple stages may also indicate at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, and the operation status of each of the multiple stages (i.e., whether the operation is completed). In some embodiments, since there are limitations on the input information of the machine learning model (e.g., the length of the input information), the operation information of multiple stages may also indicate the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages. In this way, user 105 can make the input information provided to the machine learning model meet the requirements of the machine learning model, thereby ensuring the stable operation of the machine learning model.

[0061] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include at least one predefined stage, and at least one predefined stage includes a stage for generating a database query request. The at least one predefined stage may be some debugging stages pre-set by the application management platform 110 according to the characteristics of the target application and / or the corresponding functions. The purpose of the application management platform 110 performing a test query for the target application may be to generate a better database query request to return a more accurate query result to the user. Therefore, the at least one predefined stage may also include a stage for generating a database query request, which may generate an accurate and efficient database query request based on the output of the previous predefined stage. The database query request here may be, for example, a SQL (Structured Query Language) statement.

[0062] In some embodiments, the multiple stages displayed in the staged debugging area 430 may include at least one stage determined based on a test query in addition to at least one predefined stage. After the application management platform 110 executes the database query request to obtain the corresponding data, at least one stage may also be determined based on the requirements of the test query. In one example, the test query is "What is the number of commodities with a unit price lower than XX yuan?" The test query does not have additional data analysis requirements, so the application management platform 110 may only determine one stage, for example, the data viewing stage, and the application management platform 110 may output the number of commodities with a unit price lower than XX yuan in this stage. In another example, the test query is "What is the proportion of commodities with a place of origin of XX? Draw a pie chart", and the test query has additional data analysis requirements, so the application management platform 110 may generate multiple stages, for example, a chart generation stage, a data insight stage, etc. The application management platform 110 may output a pie chart of the proportion of commodities with a place of origin of XX to the chart generation stage, and the application management platform 110 may also output the data mining and analysis of the proportion of commodities with a place of origin of XX to the data insight stage. In this way, the application management platform 110 can determine different stages after the predefined stage based on the test query, which can meet different query requirements of users and improve the query experience of users.

[0063] In some embodiments, the test query for the target application executed by the application management platform 110 is a query for the first function in the target application. Since different functions in the target application have great differences, debugging for one function in the target application may not be applicable to another function in the target application, so the test query is a query for a specific function in the target application.

[0064] In some embodiments, the predefined stages may be different for different functions in the target application. For example, taking the first function as an example, at least one predefined stage may be configured for the first function in the target application. The application management platform 110 may configure at least one different predefined stage based on the characteristics of different functions in the target application. Exemplarily, for the application data analysis function in the target application, the application management platform 110 may configure five predefined stages, namely, the recall and selection of terminology stage, the recall and selection of running reference case stage, the recall and selection of target data set stage, the recall and selection of target data set field stage, and the generation and execution of database query request stage. In the recall and selection of terminology stage, the interpretation of the terminology used in the test query may be recalled and selected. In the recall and selection of running reference case stage, the running reference case similar to the test query may be recalled and selected. In the recall and selection of target data set stage, the data set required for executing the test query may be recalled and selected. In the recall and selection of target data set field stage, the field in the data set required for executing the test query may be recalled and selected. In the generation and execution of database query request stage, a database query request may be generated and executed based on the output of the previous four stages.

[0065] The above describes the division of stages using the data analysis function as an example, but this division is only exemplary. In addition, for other functions (e.g., data update, knowledge question and answer, etc.), appropriate stage division can be performed according to the characteristics of the corresponding functions. In this way, different functions in the target application are debugged separately using test queries, which fully considers the differences between different functions and can effectively improve the service quality provided by different functions.

[0066] After presenting the corresponding first operation information of multiple stages of executing the test query, the application management platform 110 can generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query. For example, the user 105 can first click the tuned control 450 on the page 400, determine that the test query and the corresponding operation information have been tuned, and then click the save control 460 to generate the operation reference case. After the operation reference case is generated, when the application management platform 110 receives the user query of the user 145, it can refer to the operation reference case to determine the result of the user query.

[0067] In some embodiments, the application management platform 110 can determine the adjustment of the first operation information of at least one of the multiple stages based on the corresponding first operation information of the multiple stages of the test query. Then, the application management platform 110 can re-execute the test query based on the adjusted first operation information of at least one stage. The application management platform 110 can present the corresponding second operation information of the multiple stages of the re-executed test query. If a use case generation confirmation is received, the application management platform 110 can generate a reference use case for the run based on the corresponding second operation information. After the first operation information of at least one stage is adjusted, the application management platform 110 can re-execute the test query and present the corresponding second operation information of the multiple stages of the re-executed test query in the staged debugging area 430. Reference Figure 4 The application management platform 110 may receive the use case generation confirmation through the triggering of the tuned control 450 and the saved control 460 by the user 105 to generate an operation reference use case based on the second operation information.

[0068] In some embodiments, the application management platform 110 can receive terminology adjustment information from the user 105 for the first stage of multiple stages, and re-execute the test query based on the terminology adjustment information. The terminology adjustment information indicates adding a term set to the first stage or removing a term set selected in the first stage. The term set includes an explanation of the terms used by the user group of the target application. The terms used by the user group may be used within the scope of the group, but are not common terms, so it is necessary to help the machine learning model understand such terms. The explanation of such a term may include a literal explanation of the term, and may also include a common term corresponding to the term.

[0069] Exemplarily, the first stage of executing the test query may select a term set related to the test query, and the user 105 may add or remove a term set in the first stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods sold through the C2C channel?", where C2C is a special term, and the term set selected in the first stage may include an explanation of C2C, and the user 105 may add other term sets or delete the term set selected in the first stage. After adjusting the term set, the application management platform 110 may determine one or more term sets used in the first stage based on the term adjustment information, and then re-execute the test query based on the one or more term sets.

[0070] In some embodiments, the application management platform 110 may receive use case adjustment information for the second stage of the multiple stages, and re-execute the test query based on the use case adjustment information. The use case adjustment information indicates adding a reference use case to the second stage or removing the reference use case selected in the second stage. Exemplarily, the second stage of executing the test query may select a use case similar to the test query, and the user 105 may add a reference use case or remove a reference use case in the second stage. For example, the test query of the user 105 for the target application is "What is the number of goods with a unit price of less than 5,000 yuan?", and the second stage may select a reference use case similar to the test query based on similarity calculation, and the reference use case selected in the second stage may be "What is the number of goods with a unit price of less than 8,000 yuan?", and the user 105 may add other reference use cases or move the reference use case selected in the second stage. After adjusting the reference use case, the application management platform 110 may determine one or more reference use cases used in the second stage based on the use case adjustment information, and then re-execute the test query based on the one or more reference use cases.

[0071] In some embodiments, the application management platform 110 may receive data set adjustment information for the third stage of the multiple stages, and re-execute the test query based on the data set adjustment information. Exemplarily, the third stage of executing the test query may select the data set required to execute the test query, and the user 105 may add or remove data sets in the third stage. For example, the test query of the user 105 for the target application is "What is the number of goods with a unit price of less than 5,000 yuan?" The data set selected in the third stage may be a product analysis table, and the user 105 may add other data sets or delete the data set selected in the third stage.

[0072] In some embodiments, the application management platform 110 may also receive data set field adjustment information for the fourth stage among the multiple stages, and re-execute the test query based on the data set field adjustment information. Exemplarily, the fourth stage of executing the test query may select the fields of the data set required to execute the test query, and the user 105 may add fields or remove fields in the fourth stage. For example, the test query of the user 105 for the target application is "What is the quantity of goods with a unit price of less than 5,000 yuan?" The fields of the data set selected in the fourth stage are the product number field, unit price field, origin field, and production date field in the product analysis table. The user 105 may add other fields or delete the fields selected in the fourth stage. In this way, the user can adjust the various stages of the test query through a graphical interface, which facilitates the user's understanding of debugging and simplifies the debugging process of the target application.

[0073] In some embodiments, the application management platform 110 may receive a stage operation request after determining the adjustment to the first operation information of at least one stage. The stage operation request indicates that a test query is to be executed from a specified stage among multiple stages, where the specified stage is before at least one stage or is included in at least one stage, and based on the adjusted first operation information of at least one stage, the specified stage among multiple stages and the stage after the specified stage are re-executed. Exemplarily, the user 105 has adjusted the second stage, in which case the specified stage may be the first stage before the second stage, or may be the second stage. In the case where the specified stage is the first stage, the application management platform 110 may execute the first stage and the stage after the first stage (i.e., all stages) based on the adjusted first operation information of at least the second stage.

[0074] For example, refer to Figure 4 For each of the five stages, a trigger entry for starting to re-execute the stage is provided, such as trigger controls 421, 422, 423, 424, and 425. The user 105 can trigger control 420 to execute all the stages from the beginning. In the case where the designated stage is the second stage, the application management platform 110 can execute the second stage and the stages after the second stage based on the adjusted first operation information of at least the second stage. For example, continue to refer to Figure 4 , the user 105 can trigger the control 422 to execute the second stage and the stage after the second stage. In this way, the user can flexibly run different stages, and by comparing the output results of running different stages, the effectiveness of adjusting different stages can be verified. It is understandable that although Figure 4 It is shown that the corresponding stages are executed by triggering controls 421, 422, 423, 424, and 425, but this triggering method is only illustrative. User 105 can also perform corresponding triggering in other ways, for example, by user input representing natural language to trigger the execution of the corresponding stage.

[0075] In some embodiments, after re-executing the test query, the application management platform 110 may present the operation information associated with the re-execution of the specified stage and the stage after the specified stage in the stage debugging area 430. For example, if the specified stage is the second stage, the operation information associated with the re-execution of the second stage and the stage after the second stage may be presented in the stage debugging area 430.

[0076] In some embodiments, the application management platform 110 may also present data information while presenting the corresponding first operation information of the multiple stages, the data information at least indicating at least one data set used to execute the test query, and generate an operation reference case based on the corresponding first operation information of the multiple stages and the data information. Figure 4 The application management platform 110 may display data information for executing the test query in the data information display area 470. In addition, the application management platform 110 may generate an operation reference case based on the data information or an adjustment to the data information.

[0077] In some embodiments, the adjustment of the data information may include updating the at least one data set used to perform the test query. For example, a data set used to perform the test query, namely, a commodity analysis table, is displayed in the data information display area 470. Figure 4 , user 105 can trigger add control 472 to update the data set.

[0078] In some embodiments, the adjustment of the data information may include updating at least a portion of at least one data set used to perform the test query. Here, the portion of the data set may indicate a field description in the data set. For example, a data set used to perform the test query, namely, a commodity analysis table, is displayed in the data information display area 470. Figure 4 , user 105 can trigger the modification control 474 to update the field description in the data set.

[0079] In some embodiments, the data information further indicates a term set referenced for executing the test query, the term set including interpretations of terms used by a user group of the target application. Figure 5 , Figure 5 An example of a page 500 displaying data information according to some embodiments of the present disclosure is shown. The page 500 includes a term set display area 510, and the user 105 can adjust the content in the term set display area 510.

[0080] In some embodiments, the data information also indicates a conversation record between the user and the digital assistant referenced for executing the test query. Figure 5 Page 500 includes a conversation record display area 520 , and user 105 can adjust the content in the conversation record display area 520 .

[0081] In some embodiments, the data information further indicates a reference prompt word used to perform the test query. Figure 5 , the page 500 includes a reference prompt word display area 530, and the user 105 can adjust the content in the reference prompt word display area 530. By adjusting the reference prompt words, better guidance can be provided to the machine learning model, helping the machine learning model to better complete the user's task.

[0082] In some embodiments, the data information also indicates conditions for determining candidate results for the test query. Figure 5, the page 500 includes a condition area 540, and the user 105 can adjust the similarity and the number of word searches in the condition area 540. For example, in a knowledge question answering scenario, the digital assistant searches for questions similar to the user based on similarity, and provides a predetermined number of search results to the user 145 according to the number of single searches.

[0083] refer to Fig. 6A , Fig. 6A An example of a page 600 of a use case library for a specific function in a target application according to some embodiments of the present disclosure is shown. The page 600 presents a use case library, which contains reference use cases that have been generated. The user 105 triggers the detail control 602 to jump to the page of the use case details corresponding to the use case, and the reference Figure 6B . Figure 6B An example of a page 610 of use case details according to some embodiments of the present disclosure is shown. The page 610 may include a test query and corresponding first run information 614 of multiple stages.

[0084] It can be understood that, according to the embodiments described above, an operation reference use case set including one or more operation reference use cases can be generated.

[0085] In some embodiments, after the target application 120 is released, during the operation of the target application 120, the operation reference case can be used to provide services to the user 145. Exemplarily, in response to receiving a user query for the target application, the application management platform 110 can search for a test query similar to or matching the user query from the operation reference case set. Based on the similarity between the user query and a test query exceeding the threshold similarity, the application management platform 110 can determine the operation reference case as the target operation reference case for the user query, and execute the user query based on the target operation reference case. After generating the operation reference case, the application management platform 110 receives the user query for the target application initiated by the user 145, and can determine the similarity between the user query and the test query through similarity calculation, and determine the operation reference case as the target operation reference case for the user query based on the similarity. After determining the target operation reference case, the application management platform 110 can use the target operation reference case to execute the user query. In this way, the result of the user query is determined with reference to the operation reference case, and more accurate query results can be provided to the user, thereby improving the user's satisfaction.

[0086] In some embodiments, a machine learning model can be used to execute user queries. For example, the application management platform 110 can provide the determined target reference use case to the machine learning model as part of the prompt word for the user query. Then, the application management platform 110 can obtain the execution result of the user query from the machine learning model. It should be noted that the target reference use case here is selected from the set of running reference use cases generated during the application creation phase or the application testing phase. Such target reference use cases are generated for debugging purposes, rather than from use cases when users actually use the application.

[0087] In some embodiments, the user using the target application 120 may be a tester, that is, the user query may come from a tester. In this case, if the tester gives positive feedback on a query result, an operation reference case may be generated based on the user query and the corresponding query result. For example, the operation information of multiple stages of the operation process of obtaining the query result may be stored as an operation reference case. In this way, user feedback is used to enrich the source of operation reference cases.

[0088] In some embodiments, the target application includes at least one function, and the test query and the user query are both queries for a first function in the at least one function. The target application may include functions such as application data analysis, application data update, and knowledge question and answer. The test query and the user query need to be for the same function, for example, the test query and the user query are both for the application data analysis function.

[0089] During the creation or testing of the target application 120, it may be necessary to evaluate the performance of the target application 120. For example, in the case where the target application includes multiple functions, it may be necessary to evaluate one or some of the functions.

[0090] To this end, in some embodiments, the application management platform 110 may maintain a test sample set for testing the target application 120, for example, testing one or more functions of the target application 120, such as a data analysis function, a data update function, a knowledge question and answer function, etc. Each test sample in the test sample set may include a sample query and a result (also referred to as a sample query result, a second result) generated by the target application 120 for the sample query.

[0091] In some embodiments, in order to obtain the evaluation result of a function in the target application 120, a machine learning model can be used to obtain feedback on the sample query results in the evaluation sample. For example, the machine learning model can score the sample query results generated by the target application 120 from one or more dimensions.

[0092] The machine learning model can give feedback in any suitable manner. As an example, an automatic evaluation method is described. The data source (e.g., knowledge base, data set, etc.) used by the target application 120 to provide services can be provided to the machine learning model, so that the machine learning model extracts queries and corresponding query results therefrom as correct examples. For example, for the knowledge question and answer function, questions and corresponding answers can be extracted. Then, the extracted queries can be executed in the target application 120, and the execution results are generated by the target application 120. Furthermore, the machine learning model can give feedback on the execution results based on the answers and the execution results in various suitable dimensions, such as scoring. Examples of evaluation dimensions may include, but are not limited to, helpfulness to users, language fluency, factual correctness, logic, etc.

[0093] Thus, the feedback given by the machine learning model for each extracted question and execution result can determine the evaluation result of the target application 120. For example, if the evaluation sample set is used for the knowledge question and answer function, the evaluation result of the knowledge question and answer function can be obtained. In this way, the user 105 can debug the target application 120 according to the evaluation result, for example, debug one or more functions.

[0094] During the creation or testing of the target application 120, the creator or tester may input some evaluation queries. The target application 120 may generate corresponding query results for the evaluation queries, also referred to as first results or evaluation query results. Then, the creator or tester may give feedback on the evaluation query results, also referred to as evaluation feedback.

[0095] In some embodiments, the evaluation feedback can be used in conjunction with the evaluation sample set to provide a more accurate evaluation result for the target application 120. Exemplarily, the operation management platform 110 can generate an evaluation reference case for the target application 120 based on the evaluation query given by the creator or tester. The evaluation reference case includes the evaluation query from the creator or tester, the evaluation query result generated by the target application 120 for the evaluation query, and the evaluation feedback on the evaluation query result. The evaluation feedback can be positive feedback or negative feedback, and can also indicate the user 145's satisfaction with the query result.

[0096] Generally speaking, for similar queries, it is expected that the evaluation results and evaluation feedback are consistent or matched. In view of this, sample queries similar to the evaluation query can be found from the evaluation sample set, for example, the similarity exceeds the threshold similarity. The evaluation sample including such similar sample queries is also called the target evaluation sample, and the target evaluation sample also includes the sample results generated by the target application 120 for the sample query.

[0097] The application management platform 110 can generate an evaluation result for the target application 120 based on the evaluation query result, evaluation feedback and sample query result. For example, the score of the sample query result in the target evaluation sample can be matched with the evaluation feedback. It is understandable that the evaluation reference case provides examples of what kind of results the creator or tester is satisfied with or what kind of results the creator or tester is not satisfied with.

[0098] In some embodiments, as described above, the evaluation of the target application 120 can be performed using a machine learning model. Accordingly, a prompt word can be generated based on at least a portion of the evaluation reference case (e.g., evaluation feedback and evaluation query results), and the prompt word is provided to the machine learning model. The machine learning model can generate feedback for the sample query results in the target evaluation sample. For example, the machine learning model can score the sample query results. Then, based on the feedback, the evaluation results for the target application 120 can be determined. It can be understood that the evaluation reference case gives examples of what kind of results are satisfactory to the creator or tester or what kind of results are unsatisfactory to the creator or tester. In this way, the machine learning model can refer to such examples in generating feedback (e.g., scoring) for the sample results. Note that the machine learning model used in the evaluation can be the same as or different from the machine learning model used in the operation of the target application 120.

[0099] In some embodiments, if the target application 120 includes at least one function, the sample query for evaluation and the evaluation query may be queries for the same function. For example, the evaluation sample set may be a knowledge question and answer function dedicated to evaluating the target application 120. In this case, the collected user queries are also for the knowledge question and answer function.

[0100] Fig. 7A FIG. 7 is a flowchart of a process 700 for application debugging according to some embodiments of the present disclosure. The process 700 may be implemented at the application management platform 110. The process 700 is described below with reference to FIG.

[0101] In block 701 , the application management platform 110 executes a test query for a target application.

[0102] In block 702 , the application management platform 110 presents corresponding first operation information of multiple stages of executing the test query, where the operation information of the multiple stages at least indicates respective outputs of the multiple stages.

[0103] In box 703, the application management platform 110 generates an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query. The operation reference case includes the test query and the operation information of the multiple stages. The operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0104] In some embodiments, generating an operation reference case includes: determining an adjustment to first operation information of at least one stage among multiple stages; re-executing a test query based on the adjusted first operation information of at least one stage; presenting corresponding second operation information of multiple stages of the re-executed test query; and in response to receiving a case generation confirmation, generating an operation reference case based on the corresponding second operation information.

[0105] In some embodiments, re-executing a test query includes: receiving a stage running request, the stage running request indicating to execute the test query from a specified stage among multiple stages, the specified stage being before at least one stage or being included in at least one stage; based on the adjusted first running information of at least one stage, re-executing the specified stage among the multiple stages and the stage after the specified stage.

[0106] In some embodiments, presenting corresponding second execution information of the plurality of phases of re-executing the test query includes presenting execution information associated with re-execution of a specified phase and phases subsequent to the specified phase.

[0107] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0108] In some embodiments, determining adjustments to first run information of at least one of multiple stages includes: receiving term adjustment information for a first stage of the multiple stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and re-executing the test query includes: determining one or more term sets used in the first stage based on the term adjustment information; and re-executing the test query based on the one or more term sets.

[0109] In some embodiments, determining an adjustment to first operating information of at least one stage among multiple stages includes: receiving use case adjustment information for a second stage among the multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and re-executing the test query includes: determining one or more reference use cases used in the second stage based on the use case adjustment information; and re-executing the test query based on the one or more reference use cases.

[0110] In some embodiments, generating an operation reference case for a target application includes: presenting data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference case based on the corresponding first operation information of multiple stages and the data information.

[0111] In some embodiments, the data information also indicates at least one of the following: a term set referenced for executing the test query (wherein the term set includes interpretations of terms used by a user group of the target application), a conversation record between a user and a digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0112] In some embodiments, the adjustment of the data information includes at least one of the following: updating at least one data set used to execute the test query, or updating at least a portion of at least one data set used to execute the test query.

[0113] In some embodiments, process 700 also includes: in response to receiving a user query for a target application, determining that a running reference case is a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; and executing the user query based on the target running reference case.

[0114] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0115] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0116] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0117] Figure 7BFIG. 7 is a flowchart of a process 710 of information query according to some embodiments of the present disclosure. The process 710 may be implemented at the application management platform 110. The process 710 is described below with reference to FIG.

[0118] In box 711, in response to receiving a user query for a target application, the application management platform 110 determines a running reference case including the test query as a target running reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query.

[0119] At block 712 , the application management platform 110 executes the user query based on the target run reference use case.

[0120] In some embodiments, executing a user query based on a target running reference use case includes: providing the target reference use case as part of a prompt word for the user query to a machine learning model; and obtaining an execution result of the user query from the machine learning model.

[0121] In some embodiments, if the user query comes from a tester and is implemented in the application testing phase, process 710 also includes: obtaining user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generating additional running reference use cases based on corresponding execution information for executing the user query in multiple stages.

[0122] Figure 7C FIG. 7 is a flowchart of a process 720 of application evaluation according to some embodiments of the present disclosure. The process 720 may be implemented at the application management platform 110. The process 720 is described below with reference to FIG.

[0123] In block 721 , the application management platform 110 generates an evaluation reference case for the target application, wherein the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result.

[0124] In block 722 , the application management platform 110 selects a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and queries in the evaluation sample set, wherein the target evaluation sample includes a sample query and a second result generated by the target application for the sample query.

[0125] In block 723 , the application management platform 110 generates an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0126] In some embodiments, generating an evaluation result for the target application includes: providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; obtaining feedback from the machine learning model for the second result in the target evaluation sample; and determining the evaluation result for the target application based on the feedback for the second result.

[0127] In some embodiments, the target application includes at least one function, and wherein both the sample query and the user query are queries for a second function of the at least one function.

[0128] Fig. 8A A schematic structural block diagram of an apparatus 800 for application debugging according to some embodiments of the present disclosure is shown. The apparatus 800 may be implemented in or included in the application management platform 110, for example. Each module / component in the apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0129] As shown in the figure, the device 800 includes a test query execution module 801, which is configured to execute a test query for a target application. The device 800 also includes a first operation information presentation module 802, which is configured to present corresponding first operation information of multiple stages of executing the test query, and the operation information of multiple stages at least indicates the output of each of the multiple stages. The device 800 also includes an operation reference case generation module 803, which is configured to generate an operation reference case for the target application based on the corresponding first operation information of the multiple stages of the test query, and the operation reference case includes the test query and the operation information of multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

[0130] In some embodiments, the reference case generation module 803 includes: a test query re-execution module, configured to determine adjustments to first operation information of at least one of multiple stages; re-execute the test query based on the adjusted first operation information of at least one stage; present corresponding second operation information of multiple stages of the re-executed test query; and in response to receiving a case generation confirmation, generate an operation reference case based on the corresponding second operation information.

[0131] In some embodiments, the test query re-execution module includes: a stage execution module, configured to receive a stage running request, the stage running request indicating to execute the test query from a specified stage among multiple stages, the specified stage being before at least one stage or included in at least one stage; based on the adjusted first running information of at least one stage, re-execute the specified stage among multiple stages and the stage after the specified stage.

[0132] In some embodiments, the stage execution module includes: a re-execution information presentation module configured to present execution information associated with re-execution of a designated stage and a stage subsequent to the designated stage.

[0133] In some embodiments, the operation information of multiple stages also indicates at least one of the following: the input of each of the multiple stages, the operation time of each of the multiple stages, the operation log of each of the multiple stages, the operation status of each of the multiple stages, and the input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

[0134] In some embodiments, the test query re-execution module includes: a first adjustment information receiving module, configured to receive term adjustment information for a first stage among multiple stages, the term adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including interpretations of terms used by a user group of a target application, and the test query re-execution module is further configured to determine one or more term sets used in the first stage based on the term adjustment information; and re-execute the test query based on the one or more term sets.

[0135] In some embodiments, the test query re-execution module includes: a second adjustment information receiving module, configured to receive use case adjustment information for a second stage among multiple stages, the use case adjustment information indicating adding a reference case to the second stage or removing a reference case selected in the second stage, and the test query re-execution module is further configured to determine one or more reference use cases used in the second stage based on the use case adjustment information; and re-execute the test query based on the one or more reference use cases.

[0136] In some embodiments, the device 800 also includes: a data information presentation module, configured to present data information while presenting corresponding first operation information of multiple stages, the data information at least indicating at least one data set used to execute the test query; and generating an operation reference use case based on the corresponding first operation information and data information of multiple stages.

[0137] In some embodiments, the data information also indicates at least one of the following: a terminology set referenced for executing the test query, the terminology set including interpretations of terms used by the user group of the target application, a conversation record between a user and a digital assistant referenced for executing the test query, a reference prompt word for executing the test query, and conditions for determining candidate results for the test query.

[0138] In some embodiments, the data information presenting module includes: a data updating module configured to update at least one data set used to execute a test query, or to update at least a portion of at least one data set used to execute a test query.

[0139] In some embodiments, the device 800 also includes: a user query execution module, configured to, in response to receiving a user query for a target application, determine that the running reference case is a target running reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity; and execute the user query based on the target running reference case.

[0140] In some embodiments, the target application includes at least one function, and wherein the test query and the user query are both queries for a first function of the at least one function.

[0141] In some embodiments, the plurality of phases include: at least one predefined phase, the at least one predefined phase including a phase of generating a database query request; and at least one phase determined based on the test query.

[0142] In some embodiments, the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

[0143] Figure 8B A schematic structural block diagram of an apparatus 810 for information query according to some embodiments of the present disclosure is shown. The apparatus 810 may be implemented in or included in the application management platform 110, for example. Each module / component in the apparatus 810 may be implemented by hardware, software, firmware, or any combination thereof.

[0144] As shown in the figure, the device 810 includes a running reference case determination module 811, which is configured to, in response to receiving a user query for a target application, determine a running reference case including a test query as a target running reference case for the user query based on the similarity between the user query and the test query exceeding a threshold similarity, wherein the running reference case includes the test query and corresponding running information of multiple stages of the test query. The device 810 also includes a user query execution module 812, which is configured to execute the user query based on the target running reference case.

[0145] In some embodiments, the user query execution module 812 includes an execution result acquisition module, which is configured to provide the target reference use case as part of the prompt word for the user query to the machine learning model; and obtain the execution result of the user query from the machine learning model.

[0146] In some embodiments, the device 810 also includes an additional operation reference case generation module, which is configured to obtain user feedback on the execution results of the user query; and in response to the user feedback being positive feedback, generate additional operation reference cases based on corresponding execution information for executing the user query in multiple stages.

[0147] Figure 8C A schematic structural block diagram of an apparatus 820 for application evaluation according to some embodiments of the present disclosure is shown. The apparatus 820 may be implemented in or included in the application management platform 110, for example. Each module / component in the apparatus 820 may be implemented by hardware, software, firmware, or any combination thereof.

[0148] As shown in the figure, the device 820 includes an evaluation reference case generation module 821, which is configured to generate an evaluation reference case for the target application, and the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and an evaluation feedback on the first result. The device 820 also includes a target evaluation sample selection module 822, which is configured to select a target evaluation sample from the evaluation sample set based on the similarity between the evaluation query and the query in the evaluation sample set, and the target evaluation sample includes a sample query and a second result generated by the target application for the sample query. The device 820 also includes an evaluation result generation module 823, which is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback, and the second result.

[0149] In some embodiments, the evaluation result generation module 823 includes a model feedback acquisition module, which is configured to provide prompt words generated based on the first result and the evaluation feedback to the machine learning model; obtain feedback on the second result in the target evaluation sample from the machine learning model; and determine the evaluation result for the target application based on the feedback on the second result.

[0150] In some embodiments, the target application includes at least one function, and wherein both the sample query and the user query are queries for a second function of the at least one function.

[0151] Fig. 9 900 is a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. It should be understood that Fig. 9 The electronic device 900 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Fig. 9 The electronic device 900 shown may include or be implemented as Figure 1 Application management platform 110, Fig. 8A Device 800, Figure 8B The device 810, or Figure 8C Device 820.

[0152] like Fig. 9 As shown, the electronic device 900 is in the form of a general electronic device. The components of the electronic device 900 may include, but are not limited to, one or more processors or processing units 910, a memory 920, a storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 920. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 900.

[0153] The electronic device 900 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 920 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 930 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 900.

[0154] The electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Fig. 9 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. Memory 920 may include a computer program product 925 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0155] The communication unit 940 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 900 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 900 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0156] The input device 950 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 960 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 900 may also communicate with one or more external devices (not shown) through the communication unit 940 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 900, or communicate with any device that allows the electronic device 900 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0157] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0158] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0159] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0160] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0161] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0162] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A method for application debugging, comprising: Execute test queries against the target application; Presenting corresponding first operation information of a plurality of stages of executing the test query, wherein the operation information of the plurality of stages at least indicates outputs of the plurality of stages respectively; as well as Based on the corresponding first operation information of multiple stages of the test query, an operation reference case for the target application is generated, the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

2. The method according to claim 1, wherein generating the operation reference use case comprises: determining an adjustment to first operating information of at least one of the plurality of phases; re-executing the test query based on the adjusted first operation information of the at least one stage; presenting corresponding second run information of re-executing the plurality of stages of the test query; as well as In response to receiving the use case generation confirmation, the operation reference use case is generated based on the corresponding second operation information.

3. The method of claim 2, wherein re-executing the test query comprises: receiving a stage execution request, the stage execution request instructing to execute the test query from a specified stage among the plurality of stages, the specified stage being before or included in the at least one stage; as well as The designated phase and phases subsequent to the designated phase among the plurality of phases are re-executed based on the adjusted first operation information of the at least one phase.

4. The method of claim 3, wherein presenting corresponding second run information of the plurality of stages of re-executing the test query comprises: Run information associated with rerunning of the specified phase and phases subsequent to the specified phase is presented.

5. The method according to claim 1, wherein the operation information of the plurality of stages further indicates at least one of the following: The inputs of the respective stages are The operation time of each of the multiple stages is The operation logs of each of the multiple stages, The operating status of each of the multiple stages, Input information provided to the machine learning model when the machine learning model is called in at least one of the multiple stages.

6. The method of claim 2, wherein determining an adjustment to the first operating information of at least one of the plurality of phases comprises: receiving terminology adjustment information for a first stage of the plurality of stages, the terminology adjustment information indicating adding a term set to the first stage or removing a term set selected in the first stage, the term set including explanations of terms used by a user group of the target application, and Wherein re-executing the test query comprises: Based on the term adjustment information, determining one or more term sets used in the first stage; and The test query is re-executed based on the one or more term sets.

7. The method of claim 2, wherein determining an adjustment to the first operating information of at least one of the plurality of phases comprises: receiving use case adjustment information for a second stage in the plurality of stages, the use case adjustment information indicating adding a reference use case to the second stage or removing a reference use case selected in the second stage, and Wherein re-executing the test query comprises: Determining one or more reference use cases used in the second stage based on the use case adjustment information; and Based on the one or more reference use cases, the test query is re-executed.

8. The method according to claim 1, wherein generating a running reference use case for the target application comprises: Presenting data information while presenting the corresponding first operation information of the plurality of stages, the data information at least indicating at least one data set used to execute the test query; as well as The operation reference use case is generated based on the corresponding first operation information of the multiple stages and the data information.

9. The method according to claim 8, wherein the data information further indicates at least one of the following: a term set referenced for executing the test query, the term set including explanations of terms used by a user group of the target application, A record of a conversation between a user and a digital assistant referenced for executing the test query, A reference prompt word for executing the test query, A condition used to determine candidate results for the test query.

10. The method according to claim 8, wherein the adjustment of the data information comprises at least one of the following: an update to the at least one data set used to execute the test query, or An update to at least a portion of the at least one data set used to execute the test query.

11. The method according to claim 1, further comprising: In response to receiving a user query for the target application, determining the running reference case as a target running reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity; as well as A reference use case is run based on the goal to execute the user query. 12 . The method according to claim 11 , wherein the target application includes at least one function, and wherein both the test query and the user query are queries for a first function among the at least one function.

13. The method of claim 1, wherein the plurality of stages comprises: at least one predefined phase, the at least one predefined phase comprising a phase of generating a database query request; as well as At least one phase is determined based on the test query. 14 . The method according to claim 13 , wherein the test query is a query for a first function in the target application, and wherein the at least one predefined phase is configured for the first function.

15. A method for information query, comprising: In response to receiving a user query for a target application, based on a similarity between the user query and the test query exceeding a threshold similarity, determining an operation reference case including the test query as a target operation reference case for the user query, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; as well as A reference use case is run based on the goal to execute the user query.

16. The method of claim 15, wherein executing the user query based on the target running a reference use case comprises: Providing the target reference use case to a machine learning model as part of a prompt word for the user query; as well as Obtain execution results of the user query from the machine learning model.

17. A method for application evaluation, comprising: Generate an evaluation reference case for the target application, the evaluation reference case comprising an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; Based on the similarity between the evaluation query and the queries in the evaluation sample set, selecting a target evaluation sample from the evaluation sample set, the target evaluation sample including a sample query and a second result generated by the target application for the sample query; as well as An evaluation result for the target application is generated based on the first result, the evaluation feedback, and the second result.

18. The method according to claim 17, wherein generating the evaluation result for the target application comprises: Providing a prompt word generated based on the first result and the evaluation feedback to a machine learning model; Obtaining feedback for the second result in the target evaluation sample from the machine learning model; as well as Based on the feedback on the second result, an evaluation result for the target application is determined. 19 . The method according to claim 17 , wherein the target application includes at least one function, and wherein both the sample query and the user query are queries for a second function among the at least one function.

20. A device for application debugging, comprising: A test query execution module, configured to execute a test query for a target application; A first operation information presentation module is configured to present corresponding first operation information of a plurality of stages of executing the test query, wherein the operation information of the plurality of stages at least indicates the output of each of the plurality of stages; as well as An operation reference case generation module is configured to generate an operation reference case for the target application based on corresponding first operation information of multiple stages of the test query, wherein the operation reference case includes the test query and the operation information of the multiple stages, and the operation reference case is referenced in the operation of the target application to determine the result of the user query.

21. A device for information query, comprising: an operation reference case determination module, configured to, in response to receiving a user query for a target application, determine an operation reference case including the test query as a target operation reference case for the user query based on a similarity between the user query and the test query exceeding a threshold similarity, wherein the operation reference case includes the test query and corresponding operation information of multiple stages of the test query; as well as The user query execution module is configured to execute the user query by running a reference use case based on the target.

22. A device for application evaluation, comprising: An evaluation reference case generation module is configured to generate an evaluation reference case for the target application, wherein the evaluation reference case includes an evaluation query, a first result generated by the target application for the evaluation query, and evaluation feedback on the first result; A target evaluation sample selection module is configured to select a target evaluation sample from the evaluation sample set based on a similarity between the evaluation query and a query in the evaluation sample set, wherein the target evaluation sample includes a sample query and a second result generated by the target application for the sample query; as well as The evaluation result generating module is configured to generate an evaluation result for the target application based on the first result, the evaluation feedback and the second result.

23. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 19 when executed by the at least one processing unit.

24. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 19.

Citation Information

Patent Citations

  • Method and device for evaluating rewriting effect of query string of search system

    CN103207871A

  • Information recommendation method and device

    CN111382256A

  • Test case allocation method and device thereof, computer equipment and storage medium

    CN113515440A

  • Debugging applications for delivery via an application delivery server

    US20200301968A1