Information processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides an information processing method and device, a storage medium and an application program. In the embodiment of the invention, in the process of using the application program by a user, interactive operation of the user on an application interface is automatically recorded, and a user operation data sequence including user operation data is generated; when information needs to be fed back to the server, determining a time range according to an information feedback scene corresponding to the target event, thereby obtaining target user operation data within the time range; furthermore, the interactive operation on the application interface corresponding to the target user operation data is restored, the corresponding interface state screenshot is intercepted, the detailed and accurate target text description information and the selective target interface state screenshot are automatically output by utilizing the information generation model and are sent to the server side, the burden of manually inputting the feedback content by the user is reduced, and the user experience is improved. Therefore, the accuracy and the efficiency of information feedback when a user encounters a problem when using the application program are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an information processing method, device, storage medium and program product. Background Art

[0002] With the rapid development of mobile Internet, various applications (Apps), such as e-commerce, life services, and social networking, have become an indispensable part of people's daily lives. In order to continuously optimize product functions, fix potential problems, and improve user experience, developers need to obtain timely feedback from users during the use of applications. User feedback can not only help the development team understand the actual use of the product, but also provide valuable reference for future version iterations.

[0003] Currently, applications provide a dedicated problem feedback interface. When users encounter problems, they can collect relevant information (such as error prompts, operation steps, etc.), and then manually enter this information into the problem feedback interface and click Submit to provide feedback to the developer. However, this information feedback method relies on users to manually enter feedback content, and users need to spend extra time and energy to describe the specific circumstances of the problem. Especially when the problem involves complex operating procedures or technical details, users often find it difficult to accurately express how the problem occurred. Summary of the invention

[0004] Multiple aspects of the present application provide an information processing method, device, storage medium, and program product to improve the accuracy and efficiency of information feedback provided by users when they encounter problems using an application.

[0005] The embodiment of the present application provides an information processing method, comprising: during the operation of a target application, in response to multiple interactive operations of a user on at least one application interface, generating a user operation data sequence, wherein the user operation data sequence includes user operation data on a target user behavior path, and the at least one application interface is an application interface that appears on the target user behavior path; if a target event that requires information feedback to a server occurs during the operation of the target application, determining a start time and an end time corresponding to the target event on the target user behavior path according to an information feedback scenario corresponding to the target event; and obtaining at least one target user operation data between the start time and the end time from the user operation data on the target user behavior path included in the user operation data sequence. The invention relates to a method for generating an information generation model based on artificial intelligence (AI), wherein the at least one target user operation data and the at least one interface state screenshot are input into an information generation model based on artificial intelligence (AI), and target text description information for information feedback is generated based on the at least one target user operation data, and a target interface state screenshot for information feedback is determined based on the at least one interface state screenshot and the at least one target user operation data; the target text description information and the target interface state screenshot are reported to the server end, so that the server end can perform information analysis on the target application based on the target text description information and the target interface state screenshot.

[0006] An embodiment of the present application also provides an electronic device, including: a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the processor can implement each step of the information processing method provided in the embodiment of the present application.

[0007] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement each step of the information processing method provided in the embodiment of the present application.

[0008] The embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step of the information processing method provided in the embodiment of the present application.

[0009] In an embodiment of the present application, an information processing method is provided, which automatically records the user's interactive operations on the application interface during the user's use of the application, and generates a user operation data sequence including user operation data; when information feedback is required to be sent to the server, the time range is determined according to the information feedback scenario corresponding to the target event, so as to obtain the target user operation data within the time range; further, the interactive operations on the application interface corresponding to the target user operation data are restored and the corresponding interface status screenshots are captured, and the information generation model is used to automatically output detailed and accurate target text description information and selective target interface status screenshots, and send them to the server, thereby reducing the burden of users manually entering feedback content, thereby improving the accuracy and efficiency of information feedback when users encounter problems when using the application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A flowchart of an information processing method provided for an exemplary embodiment of the present application;

[0012] Figure 2 A schematic diagram of the structure of an application interface provided for an exemplary embodiment of the present application;

[0013] Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0015] The existing method of providing information feedback when users encounter problems using applications (Apps) relies on users manually inputting feedback content. Users need to spend extra time and energy to describe the specific circumstances of the problem. In particular, when the problem involves complex operating procedures or technical details, it is often difficult for users to accurately express the technical problems of the problem's occurrence process. In an embodiment of the present application, an information processing method is provided, which automatically records the user's interactive operations on the application interface during the user's use of the application, and generates a user operation data sequence including user operation data; when information feedback is required to be provided to the server, a time range is determined according to the information feedback scenario corresponding to the target event, thereby obtaining the target user operation data within the time range; further, the interactive operations on the application interface corresponding to the target user operation data are restored and the corresponding interface status screenshots are captured, and the information generation model is used to automatically output detailed and accurate target text description information and selective target interface status screenshots, and send them to the server, thereby reducing the burden of users manually inputting feedback content, thereby improving the accuracy and efficiency of information feedback provided by users when they encounter problems using the application.

[0016] The following is combined with Figure 1 , describe in detail the technical solutions provided by each embodiment of the present application.

[0017] Figure 1 The exemplary embodiment of the present application provides a flowchart of an information processing method. Figure 1 As shown, the method includes:

[0018] S101, during the operation of the target application, responding to multiple interactive operations of the user on at least one application interface to generate a user operation data sequence;

[0019] S102: If a target event that needs to provide information feedback to the server occurs during the operation of the target application, determine the start time and end time corresponding to the target event on the target user behavior path according to the information feedback scenario corresponding to the target event;

[0020] S103, acquiring at least one target user operation data between a start time and an end time from the user operation data on the target user behavior path included in the user operation data sequence;

[0021] S104, restoring at least one interactive operation of the user on the corresponding application interface based on at least one target user operation data, and capturing at least one interface state screenshot corresponding to the at least one interactive operation during the restoration process;

[0022] S105, inputting at least one target user operation data and at least one interface state screenshot into an information generation model based on artificial intelligence AI, generating target text description information for information feedback based on the at least one target user operation data, and determining a target interface state screenshot for information feedback based on the at least one interface state screenshot and the at least one target user operation data;

[0023] S106: Report the target text description information and the target interface status screenshot to the server.

[0024] In the embodiments of the present application, the application type of the target application is not limited, and any software program or service that a user can interact with can be regarded as the target application in the embodiments of the present application. For example, the target application can be a social media application, an online shopping platform, a game application, a housekeeping service application, a transportation application, etc.

[0025] In the embodiment of the present application, the application interface on the target application refers to the visualization and operability part of the user's interaction with the application program composed of a series of design elements, and each application interface includes different components, such as buttons, text boxes, menus, icons, etc. The user can interact with the components on the application interface in a variety of forms, and the function or state change in the target application can be triggered by the interactive operation. The embodiment of the present application does not limit the specific form of the interactive operation. For example, the form of the interactive operation can be that the user directly clicks (including single click, double click), slides (including up, down, left, right slide), long presses, zooms (zoom in, zoom out) and other operations on the component on the touch screen with a finger, or the user inputs text, numbers or other symbols through a physical keyboard or a virtual keyboard on the screen, or the user operates through physical buttons on the device, such as a power button, a volume button, a shutter button, etc.

[0026] In the embodiment of the present application, the concepts of user behavior path and target user behavior path are predefined. User behavior path refers to the path formed by a series of operations performed by the user from the beginning to the end in the target application. The user behavior path describes the order and manner in which the user performs interactive operations in the target application, which is specifically manifested as the different application interfaces accessed by the user and the interactive operations performed on these interfaces. For example, the user is using an online shopping application to search for and purchase goods, and the user behavior path is: 1. Open the application 2. Search for goods 3. Click on the search results 4. View the details of the goods 5. Add the goods to the shopping cart 6. Check out the shopping cart 7. Fill in the delivery address 8. Complete the payment. At least part of the path that can be participated by the user during the operation of the target application is used as the target user behavior path. The target user behavior path can be flexibly defined according to different target applications, and there is no limitation on this. During the operation of the target application, in response to multiple interactive operations of the user on at least one application interface, a user operation data sequence is generated. The user operation data sequence includes user operation data on the target user behavior path, and the at least one application interface here is an application interface that appears on the target user behavior path. The target application records each interactive operation of the user to form a sequence, which is called the user operation data sequence. The user operation data sequence includes user operation data, which are arranged in chronological order and can fully reflect the interactive operation between the user and the target application and the behavioral path of the interactive operation. The user operation data includes the operation name of the interactive operation, the logo of the application interface where the interactive operation is located, the UI (User Interface) elements (such as pictures, links, drop-down lists, controls, etc.) affected by the interactive operation, the environmental data of the interactive operation (such as information about hardware devices, network type and network status, geographic location, etc.), and the time when the operation occurs.

[0027] Figure 2 This is a schematic diagram of the structure of the application interface provided by the exemplary embodiment of the present application. Figure 2As shown, application interface 1 includes function module 1, function module 2, and function module 3. The user clicks position 1 to enter the application interface of function module 1, i.e., application interface 2. If function module 1 is a house renting function module, the user clicks position 1 to jump into application interface 2 about the house renting function module. Application interface 2 includes function module 4, function module 5, and function module 6. The function module to which application interface 2 belongs is function module 1 of application interface 1. The user operation data of the interactive operation of clicking position 1 is "operation name: click, identification of the application interface where the interactive operation is located: identification of application interface 1, UI element affected by the operation: function module 1; data involved in the operation: none; environmental data of the operation: mobile phone model, WiFi (Wireless Fidelity, wireless network communication technology), good network status, xxx location, operation time: xxxx year-xx month-xx day-xx time". For example, a user is using an online shopping application, and the user's interactive operation is to enter "car model A" in the input box on the application homepage to search. The user operation data of this operation is "operation name: input, identification of the application interface where the interactive operation is located: identification of the application homepage, UI element affected by the operation: homepage input box; data involved in the operation: car model A; environmental data of the operation: mobile phone model, WiFi (Wireless Fidelity, wireless network communication technology), good network status, xxx location, operation time: xxxx year-xx month-xx day-xx time".

[0028] In addition, if the user performs interactive operations on the application interface of the target application, which involves network interface requests, that is, when the target application communicates with the server or other remote services, the data involved in the user interactive operations include not only the data involved in the interactive operations on the application interface, but also interface request parameters, interface request data, interface response data, timestamp of the interface request, and context information of the interface request. The target application generates a complete user operation data sequence based on the user's interactive operations and network interface requests.

[0029] In the embodiment of the present application, there is no limitation on the storage form of user operation data. The user operation data sequence including the user operation data on the target user behavior path is all stored locally in the target application, and different storage methods can be selected according to specific needs and application scenarios. For example, user operation data can be stored in a local file (such as a text file or a binary file to store data), or in a local database (such as a lightweight relational database, a non-relational database, etc.).

[0030] In the embodiments of the present application, the type of target event is not limited. The target event refers to an event that occurs during the operation of the application and needs to be fed back to the server for further analysis. For example, the target event can be a user actively reporting a problem, or a target application crash (i.e., the target application suddenly stops running, causing the user to be unable to continue using it), or a target application that responds slowly, freezes, or other situations that affect the user experience, or even a system error prompt (i.e., the target application detects an internal error or anomaly and displays a corresponding error prompt to the user).

[0031] In an embodiment of the present application, the start time and end time corresponding to the target event on the target user behavior path are determined according to the information feedback scenario corresponding to the target event. Among them, the information feedback scenario refers to a specific scenario associated with the target event. The information feedback scenario is divided into product improvement type and functional abnormality type. Each target event will correspond to one of these two information feedback scenarios. When the target event occurs, the target application determines a specific time period according to the information feedback scenario. The user behavior path within this time period is directly related to the target event. The start time point and end time point of this time period are the "start time" and "end time" that need to be determined. By determining the start time and end time corresponding to the target event on the user behavior path, the time when the problem occurs can be located more accurately, which helps to narrow the scope of the problem investigation, reduce the interference of irrelevant data and the amount of data analyzed, thereby improving the accuracy and efficiency of information feedback by the target application.

[0032] In an embodiment of the present application, at least one target user operation data located between the start time and the end time is obtained from the user operation data on the target user behavior path included in the user operation data sequence. Among them, at least one target user operation data is extracted from the user operation data sequence, and is a user behavior record directly related to the target event. These target user operation data can be referred to as user operation data segments, and the user operation data segments contain the target user operation data of the user between the above-mentioned start time and the end time. The user operation data segment can contain the user operation data of a single interactive operation, or it can be a collection of user operation data of multiple interactive operations, depending on how many interactive operations the user has performed between the start time and the end time (i.e., the user's behavior complexity and operation frequency). If the user only performs one interactive operation between the start time and the end time, then the user operation data segment only contains one target user operation data; if the user performs multiple interactive operations between the start time and the end time, then the user operation data segment will contain multiple target user operation data. For example, after the user clicks the "Submit Order" button on the checkout interface, he immediately encounters a payment failure. At this time, the user operation data segment only contains one target user operation data, that is, data information about the time and operation results of the user clicking the "Submit Order" button; when the user fills in the order information, he selects the delivery address, payment method, and coupon in turn, and modifies the quantity of goods several times, and then encounters a payment failure after clicking the "Confirm Payment" button. At this time, the user operation data segment will contain multiple target user operation data, covering all the user's interactive operations during this period. By accurately locating and extracting target user operation data, it can not only improve the accuracy of information feedback and reduce the workload of user feedback, but also enhance the ability of subsequent recovery paths.

[0033] In an embodiment of the present application, at least one interactive operation of a user on a corresponding application interface is restored based on at least one target user operation data. Restoring the interactive operation means reconstructing the interactive operation performed by the user on the application interface. This process simulates a series of interactive operations of the user in a "playback" manner to reproduce the problems encountered during the user operation. In the process of restoring the user interactive operation, the system will capture a screenshot of the interface state corresponding to at least one interactive operation. The interface state screenshot is a "snapshot" of the application interface when the user performs the interactive operation. The interface state screenshot is used to record the changes in the application interface before and after the user performs the interactive operation, including the displayed content, layout, prompt information, etc. For example, when the user fills in the order information, the system will respectively capture the interface states corresponding to the operations such as selecting the delivery address, selecting the payment method, and clicking the "Confirm Payment" button. By restoring the user operation and capturing the interface operation screenshot, intuitive visual evidence is provided to help developers analyze and solve problems more efficiently, while reducing the workload of users in providing information feedback and improving the user experience.

[0034] In an embodiment of the present application, at least one target user operation data and at least one interface state screenshot are input into an information generation model based on artificial intelligence AI. Among them, the input method for at least one target user operation data and at least one interface state screenshot can be divided into multiple inputs, that is, at least one target user operation data and at least one interface state screenshot are divided into two inputs into the information generation model based on artificial intelligence AI. The multiple input method is suitable for scenarios where different types of data need to be processed independently, or when the amount of data is large, batch processing can improve the efficiency of the system; the input method can also be a single input, that is, at least one target user operation data and at least one interface state screenshot are simultaneously input into the information generation model, and two operations are performed inside the information generation model. The single input method is suitable for scenarios where user interaction operations and interface states need to be comprehensively analyzed to ensure the correlation between the target user operation data and the interface state screenshots.

[0035] Furthermore, the information generation model based on artificial intelligence AI can understand and analyze the input user operation data and interface status screenshots. On the one hand, the information generation model generates target text description information for information feedback based on at least one target user operation data. The target text description information reflects the specific operation process and problems encountered by the user when the problem occurs, including first, a detailed description of the user's interactive operations on the application interface, such as the buttons clicked, the information entered, the options selected, etc., and records the timestamp of each interactive operation; second, a description of the specific problems encountered by the user, such as the crash of the target application, the failure of the function or the error prompt; in addition, other information related to the problem is provided, such as the user's device type, the operating system version and the network environment. In the above manner, the target text description information can not only clearly display the user's operation path, but also comprehensively record the problems encountered by the user and their background information, helping the development team to reproduce and solve the problem more accurately. On the other hand, the information generation model determines the target interface status screenshot for information feedback based on at least one interface status screenshot and at least one target user operation data. The target interface status screenshot may be one or more interface status screenshots that can represent the scene where the problem occurs, so as to provide more comprehensive visual evidence. The information generation model based on artificial intelligence (AI) generates target text description information for information feedback and determines the target interface status screenshot for information feedback, which improves the accuracy and efficiency of information feedback and reduces the burden of users manually entering information feedback.

[0036] In the embodiment of the present application, the target text description information and the target interface status screenshot are reported to the server, so that the server can analyze the target application information based on the target text description information and the target interface status screenshot. The server is not limited to a single physical server or virtual machine. It can be a cluster composed of multiple servers, or a service provided by a cloud computing platform. As long as it can provide stable data storage and processing capabilities and support large-scale concurrent access, the system or platform can be used as a server, and the embodiment of the present application does not limit it.

[0037] In the embodiments of the present application, the implementation method of generating the user operation data sequence in step S101 is not limited. In an optional embodiment, a stubbing code can be pre-implanted on the target user behavior path of the target application, and then a user operation data sequence is generated in response to multiple interactive operations of the user on at least one application interface, including: sequentially running the stubbing code implanted on the target user behavior path, sensing the user's interactive operations on the corresponding application interface through the stubbing code, and recording the detailed information of the interactive operations in sequence to obtain a user operation data sequence. Among them, the user operation data sequence not only includes all the interactive operations of the user on the application interface, but also retains the user operation data associated with the interactive operations. In order to accurately record the user's interactive operations in the target application, the system will pre-implant the stubbing code on the target user behavior path. The stubbing code refers to a code snippet inserted at a key position of the target application to capture and record the user's interactive operations. The stubbing code can be implanted by the developer at the expected user interaction operation point when developing the target application; it can also be added by the developer in the final code review and testing stage before the target application is released; it can also be dynamically implanted into the target application through plug-ins, patches or upgrades after the target program is released. The embodiment of the present application does not limit the implantation time of the stubbing code. If the user behavior path of the target application is simple, a stubbing code can be implanted at the key entry point to capture the interactive operations of the user on the target user behavior path during the interaction with the target application; if the user behavior path of the target application is complex, especially for applications with multiple interfaces or multiple functional modules, a stubbing code can be implanted on each interface or each functional module, and each stubbing code is only responsible for capturing the interactive operations within the interface or the functional module to ensure the accuracy of the user operation data.

[0038] In an optional embodiment, in step S102, a target event that requires information feedback to the server occurs during the operation of the target application, and the start time and end time corresponding to the target event on the target user behavior path are determined according to the information feedback scenario corresponding to the target event, including: if a target event that requires information feedback to the server occurs during the operation of the target application, the information feedback scenario corresponding to the target event is determined to be a product improvement scenario or a functional abnormality scenario; if the information feedback scenario is a product improvement scenario and the target event is a user-triggered event, the occurrence time of the user-triggered event is obtained as the end time, and the target function module to which the application interface where the user-triggered event is located is determined, and the time of entering the target function module is used as the start time.

[0039] Among them, the product improvement scenario refers to the user discovering that the function or interface of the target application can be optimized during the use of the target application, thereby actively triggering the target application to automatically provide information feedback; and the user-triggered event refers to the operation or behavior initiated by the user, which causes the target application to provide information feedback to the server. For example, in the process of using the housekeeping APP, the specific operations are as follows: the user clicks the "housekeeping service" button from the homepage and enters the nanny service selection interface at 10:00:10; the user enters the keyword "nanny" in the search box, selects the service area as "xx area", and then clicks the "search" button at 10:00:15; the user browses the search result list and views the detailed information of a nanny between 10:00:20 and 10:00:30. During the browsing process, the user found that the page was stuck when viewing the nanny's detailed information, and triggered automatic information feedback at 10:00:40. The target application records the entire operation path of the user from entering the nanny service selection interface to submitting feedback, and recognizes that the user submits feedback on the nanny details information interface. Therefore, it is determined that the nanny service selection interface belongs to the "housekeeping service" function module. The time when the user enters the nanny service selection interface is 10:00:10, which is used as the start time. At the same time, the time when the user triggers automatic information feedback is recorded, which is 10:00:40, and it is used as the end time.

[0040] Optionally, if the information feedback scenario is a functional abnormality scenario, the time when the target application has an abnormality is obtained as the end time, and the target application interface where the abnormality occurs is determined, and the time when the target application interface is entered is used as the start time. Among them, the functional abnormality scenario refers to the process in which the user uses the target application, and the target application has a functional problem (such as crash, error prompt, operation failure, etc.) during operation, causing the system to automatically detect the abnormality and trigger information feedback. For example, a user encountered a problem of failed reservation for maintenance service while using the housekeeping app. The specific path is as follows: the user clicked the "Maintenance Service" button from the homepage and entered the maintenance service selection interface at 10:00:10; the user entered the keyword "air conditioning maintenance" in the search box, selected the service area as "xx area", and then clicked the "Search" button at 10:00:15; the user browsed the search result list, selected a maintenance technician, and clicked the "Book Now" button at 10:00:25; the user was guided to the reservation confirmation interface and filled in the service time and contact information at 10:00:35; after the user clicked the "Submit Order" button, the interface displayed an error message "Reservation failed, please try again later" at 10:00:45. During this process, the system automatically detected the functional abnormality of the failed reservation, determined it as a functional abnormality scenario and triggered information feedback. According to the functional abnormality scenario, the system determines the specific time when the reservation failure occurred, that is, 10:00:45, as the end time. Since the user encounters the problem on the appointment confirmation interface, the time when the user enters the appointment confirmation interface, 10:00:35, is determined as the start time.

[0041] In addition, there is a special case in the functional abnormality scenario: the interactive operation that causes the functional abnormality is not only caused by the current application interface, but also because the previous interface of the current application interface has unfinished tasks. In this special case, the time when the target application abnormality occurs is obtained as the end time, and the target functional module to which the application interface where the user triggers the event belongs is determined, and the time of entering the target functional module is used as the start time. By expanding the time range and thus the target user operation data range, it is ensured that the entire process of restoring the abnormality when the user performs the interactive operation can be completed, thereby improving the accuracy of information feedback.

[0042] In an optional embodiment, if a target event that requires information feedback to the server occurs during the operation of the target application, the information feedback scenario corresponding to the target event is determined to be a product improvement scenario or a functional abnormality scenario, including: if the user initiates the first specified interactive operation during the operation of the target application, it is determined that the target event occurs, the user initiates the first interactive operation to determine that the target event is a user-triggered event, and in the case where the target event is a user-triggered event, the information feedback scenario corresponding to the target event is determined to be a product improvement scenario, and the first specified interactive operation represents the need to provide information feedback to the server. Among them, the first specified interactive operation refers to a specific interactive operation initiated by the user in the target application, which is used to trigger information feedback. The first specified interactive operation can be to trigger information feedback by shaking the device, or by long pressing the feedback button or icon set on the application interface to trigger information feedback, or by triggering information feedback through a specific gesture (such as sliding inward from the edge of the mobile phone screen), and the embodiments of the present application do not limit this.

[0043] Optionally, if an exception occurs in the target application during operation, or the user initiates a second specified interactive operation after the target application is restarted due to an exception, it is determined that a target event occurs, and the target event is an event in which an exception occurs in the target application. In the case where the target event is an event in which an exception occurs in the target application, the information feedback scenario corresponding to the target event is determined to be a functional abnormality scenario. Among them, the second specified interactive operation refers to a specific interactive operation initiated by the user after the target application is abnormal or restarted, which is used to trigger information feedback. The method of the second specified interactive operation refers to the first specified interactive operation method in the above embodiment, which will not be repeated here. For example, when the user is stuck in the target application for payment, but does not exit the target application or restart due to the stuck, the system automatically detects that the target application is abnormal and automatically provides information feedback; or, when the user is stuck in the target application for payment, and causes the application to be exited, the user needs to click the icon of the target application again to restart, and after starting the application, the information feedback is triggered by shaking the phone (i.e., the second specified interactive operation), thereby determining that the target event is an event in which an exception occurs in the target application, and determining that the information feedback scenario is a functional abnormality scenario. By determining the information feedback scenario according to the target event, the pertinence and efficiency of the information feedback are improved, which helps to improve the quality of the target application and user experience.

[0044] In an optional embodiment, at least one interactive operation of the user on the corresponding application interface is restored based on at least one target user operation data, and at least one interface state screenshot corresponding to at least one interactive operation is captured during the restoration process, including: calling an automated restoration tool, setting environmental conditions according to the operating environment data contained in at least one target user operation data, and simulating the launch of the target application. Among them, the automated restoration tool is integrated in the driver of the terminal device, and is a tool for simulating user interactive operations. The automated restoration tool can simulate the user's device environment to ensure that the device model, operating system version, application version number, network status, etc. used in the restoration process are completely consistent with the environment when the user operates, avoiding the situation where problems caused by environmental differences cannot be reproduced. For example, if an exception occurs on a mobile phone with an Android 11 system and a screen resolution of 1080x1920, the automated restoration tool will simulate the same device environment during the restoration process to ensure that the application runs under the same conditions; if the user encounters a problem under a Wi-Fi network, the automated restoration tool will also simulate the Wi-Fi network environment instead of using other types of network connections. After setting the environmental conditions, the automated restoration tool will simulate the launch of the target application.

[0045] In an embodiment of the present application, after the automated recovery tool simulates the start of the target application, each target user operation data is traversed in the order between at least one target user operation data; for the currently traversed target user operation data, the application interface to which the currently traversed target user operation data belongs and the operation action in the currently traversed target user operation data and its corresponding UI element identifier are determined, and the operation action is performed on the UI element corresponding to the UI element identifier on the application interface. Among them, the automated recovery tool will process each target user operation data one by one in the order between the target user operation data. This means that the automated recovery tool will execute each step recorded in the target user operation data in sequence according to the order of the user's actual operation. In order to ensure that the specific UI element on the application interface to which the target user operation data belongs can be accurately found and operated, each target user operation data also contains a UI element identifier, which is a value defined by the developer and is used to uniquely identify the identifier of a specific UI element on the application interface, such as an ID, a class name, or an XPath (XML Path Language) expression. The UI element identifier enables the automated tool to accurately locate and identify specific elements on the application interface. After determining the identifier of the UI element, the automated recovery tool will find the corresponding UI element on the corresponding application interface and execute the operation recorded in the target user operation data. For example, if the interactive operation is a click, after finding the button or link corresponding to the UI element identifier, the automated recovery tool will simulate the user clicking the UI element.

[0046] In the above embodiment, while performing an operation on the UI element corresponding to the UI element identifier on the application interface, the screenshot API (Application Programming Interface) provided by the operating system is called to take a screenshot of the current application interface to obtain a screenshot of the interface status corresponding to the target user operation data currently traversed. Among them, the screenshot API refers to a functional interface provided by the operating system for capturing the application interface image on the current screen. By calling the screenshot API, all visual elements on the application interface, including buttons, input boxes, texts, pictures, etc., are captured to obtain a real-time screenshot of the application interface, and it is saved as an image file or further processed. When the automated recovery tool simulates interactive operations such as user clicking buttons, entering text, sliding the screen, etc., the screenshot API provided by the operating system is called to immediately capture the current application interface status to ensure that the interface status of each user interactive operation is fully recorded. For example, the user operation data indicates that the search button is clicked, and the screenshot API is called immediately after the click event occurs to capture the interface status of the button being pressed.

[0047] In an optional embodiment, after determining the application interface to which the currently traversed target user operation data belongs, it also includes: if the application interface to which the currently traversed target user operation data belongs is different from the application interface to which the last traversed target user operation data belongs, then a page jump protocol is constructed, and based on the page jump protocol, the application interface is automatically jumped to the application interface to which the currently traversed target user operation data belongs. Among them, the application interface refers to the specific page or view where the user is operating, such as the order management page, the product search page, the settlement page, etc. In order to ensure that the automated recovery tool can accurately simulate the user's operation path, the system needs to keep track of the current application interface and compare it with the interface to which the last operation data belongs. If the application interface to which the currently traversed target user operation data belongs is different from the application interface to which the last traversed target user operation data belongs, it means that the user has a page jump in actual operation.

[0048] In an embodiment of the present application, in order to realize automatic jump from one application interface to another application interface, a page jump protocol will be constructed. Among them, the page jump protocol is a set of instructions or rules used to instruct the system on how to jump from the current interface to the target interface. Because the jump logic between different interfaces may be different, the details involved, such as the jump method, parameter passing, waiting time, etc. may also be different, so each application interface corresponds to a unique page jump protocol to ensure the accuracy and consistency of the jump process. For each application interface, it is only necessary to construct a page jump protocol once. After that, when it is necessary to jump to the application interface, the page jump protocol that has been constructed can be directly used. There is no need to repeat the construction, which avoids repeated calculations and waste of resources, and improves the operating efficiency of the system. For example, the jump method of the page jump protocol can be a navigation bar jump to the target interface by clicking a navigation bar button or link on the application interface; it can also be a return or forward jump to the previous or next page by simulating a user clicking a "back" or "forward" button; it can also be a direct jump to the target interface by calling an API or routing mechanism inside the application; it can even be a display jump that triggers a pop-up window or modal box, and the embodiments of the present application do not limit this.

[0049] In an optional embodiment, at least one target user operation data and at least one interface state screenshot are input into an information generation model based on artificial intelligence AI, target text description information for information feedback is generated based on at least one target user operation data, and the target interface state screenshot for information feedback is determined based on at least one interface state screenshot and at least one target user operation data, including: inputting at least one target user operation data into the information generation model, generating target text description information for information feedback based on at least one target user operation data; inputting at least one target user operation data and at least one interface state screenshot into the information generation model, and selecting the target interface state screenshot from at least one interface state screenshot based on at least one interface state screenshot and at least one target user operation data. Among them, the information generation model based on AI analyzes the target user operation data and automatically generates a concise and accurate target text description information, and the information generation model describes each step of the user's operation on the application interface according to the time sequence of the user's operation, for example, the user first clicks the "shopping cart" button, then enters the checkout page, and finally clicks the "confirm payment" button.

[0050] In an embodiment of the present application, while the information generation model generates target text description information for information feedback based on at least one target user operation data, the information generation model will describe in detail the results of each operation, especially operations that cause functional abnormalities or problems, such as when a user encounters an error message that payment failed after clicking the "Confirm Payment" button. If the user encounters a problem during an interactive operation, the information generation model will generate a detailed description of the specific manifestation of the problem, such as an error message that "network connection timed out" is displayed when payment fails. In addition, the information generation model can also make some preliminary improvement suggestions based on the user's operation path, such as optimizing the payment process, adding a payment retry mechanism, or improving the stability of the network connection. In this way, not only can clear and concise target text description information be generated, but also valuable debugging and optimization references can be provided to developers.

[0051] In the embodiment of the present application, the specific method for the information generation model to determine the target interface status screenshot in step S105 is not limited. In an optional embodiment, the AI-based information generation model combines the target user operation data and the interface status screenshot to determine the target interface status screenshot that reflects the problem encountered by the user. Among them, the screening criteria for determining the target interface status screenshot include the following: the information generation model gives priority to screenshots related to key operations. For example, if the user encounters a payment failure problem on the payment page, the information generation model selects a screenshot of the payment failure page instead of other irrelevant interface screenshots. The information generation model will also select interface status screenshots that can clearly show the specific manifestation of the problem, such as the error prompt displayed when the payment fails, to help the development team understand the problem more intuitively. In addition, the information generation model analyzes whether the screenshot can reflect the user's operation path. For example, if the user fills in the delivery address before the payment fails, the information generation model will select a screenshot containing the delivery address input box. In order to avoid information redundancy caused by too many screenshots, the information generation model selects the most representative screenshots to avoid repeated or unnecessary screenshots. For example, if the user performs multiple similar operations on the same application interface, the model will only select one of the screenshots. Through the screening method of the above information generation model, at least one target interface status screenshot that best reflects the problem can be selected from multiple interface status screenshots, helping the development team to more intuitively understand the user's operation path and the problems encountered.

[0052] Optionally, determining the target interface state screenshot can not only rely on directly screening the interface state screenshots according to the target user operation data, but also generate the target interface state screenshot through an information generation model based on the text description content combined with the interface state screenshots after preliminary screening. The information generation model performs preliminary screening on at least one interface state screenshot, excludes screenshots that are obviously unrelated to the problem or functional abnormality, and narrows the range of candidate interface state screenshots. Then, the information generation model generates a concise and accurate text description information based on the target user operation data, and generates the target interface state screenshot in combination with at least one interface state screenshot preliminarily selected. In this way, interface state screenshots can be selected or generated more intelligently, providing the most intuitive visual evidence, thereby improving the accuracy of information feedback.

[0053] In an optional embodiment, at least one target user operation data is input into the information generation model, and target text description information for information feedback is generated based on the at least one target user operation data, including: inputting at least one target user operation data and the information feedback scenario into the information generation model, integrating the key information in the at least one target user operation data to obtain initial text description information, the initial text description information describes the user's operation path and problem information; according to the text format and basic description information corresponding to the information feedback scenario, the initial text description information is formatted and optimized to obtain intermediate text description information; and the intermediate text description information is added with classification labels adapted to the information feedback scenario to obtain the target text description information. Among them, the information generation model includes a text processing layer, a feature extraction layer, a semantic processing layer, and a content generation layer. The target user operation data and the information feedback scenario are input into the text processing layer of the information generation model, the key information in the operation data is integrated, and the initial text description information describing the user's operation path and problem information is generated.

[0054] For example, the initial text description information is an error message that the user encountered a payment failure after clicking the "Confirm Payment" button. In the feature extraction layer of the information generation model, the key features in the initial text description information are extracted and initially optimized to ensure that the text is concise and clear, while providing a basis for subsequent processing. In the semantic processing layer of the information generation model, the format of the initial text description information is corrected and the content is optimized according to the requirements of the information feedback scenario. Different scenarios require different text formats and tones. The information generation model adjusts the text format and supplements the necessary background information or suggestions to generate intermediate text description information. For example, in the feedback of payment failure, the information generation model adds "Please check whether your network connection is stable." The content generation layer of the information generation model adds classification tags adapted to the information feedback scenario to the optimized intermediate text description information to obtain the target text description information. Classification tags help developers quickly locate the problem type, priority, involved functional modules, and device information. For example, classification tags can include "payment problem", "high priority", "Android device", etc. The target text description information finally generated not only clearly describes the problems that occur during user interaction, but also provides clear classification information, which improves the accuracy and efficiency of information feedback, thereby facilitating developers to efficiently process information feedback.

[0055] In an optional embodiment, at least one target user operation data and at least one interface state screenshot are input into an information generation model, and based on the at least one interface state screenshot and the at least one target user operation data, a target interface state screenshot is selected from the at least one interface state screenshot, including: inputting the at least one target user operation data and the at least one interface state screenshot into the information generation model, extracting the interface content in the interface state screenshot corresponding to each target user operation data, and calculating the correlation between each interface content and the corresponding target user operation data; according to the correlation between each interface content and the corresponding target user operation data, an interface state screenshot that meets the set correlation condition is selected from at least one interface state screenshot as the target interface state screenshot. Wherein, the interface content refers to the specific information contained in the interface state screenshot, such as button text, input box content, error prompt, page layout, etc. The information generation model evaluates the similarity and consistency between the interface content and the data operation data by calculating the matching degree between them, that is, by comparing the content in the interface state screenshot with the user operation data. The higher the correlation, the more accurately the interface state screenshot can reflect the problems or anomalies encountered by the user during the interactive operation. Setting a relevance condition refers to a preset threshold or standard for screening interface status screenshots that are highly relevant to user operation data. Only interface status screenshots that meet the set relevance conditions can be selected as target interface status screenshots. Through the above process, a target interface status screenshot that can reflect user problems can be selected from at least one interface status screenshot, reducing the workload of manually screening screenshots, which not only improves the accuracy of information feedback, but also improves the efficiency of developers in processing information feedback content.

[0056] In the embodiment of the present application, the information generation model is a multimodal artificial intelligence model that can process and analyze text data (such as user operation data, initial text description information, etc.) and image data (such as interface status screenshots). Through multimodal capabilities, the information generation model can integrate user operation data and information in interface screenshots to generate a more comprehensive and accurate feedback report, and select the target interface status screenshot that best reflects the user's problem.

[0057] In an optional embodiment, the information generation model further includes: a text processing layer, a feature extraction layer, an image processing layer, and a relevance calculation layer; inputting at least one target user operation data and at least one interface state screenshot into the information generation model, extracting the interface content in the interface state screenshot corresponding to each target user operation data, and calculating the relevance between each interface content and the corresponding target user operation data, including: inputting at least one interface state screenshot into the image processing layer, extracting the visual features and semantic features in at least one interface state screenshot, and generating at least one interface content according to the visual features and semantic features of at least one interface state screenshot. Among them, the image processing layer is responsible for extracting the visual features and semantic features in the interface state screenshot. The visual features include the layout and style of visual elements such as buttons, input boxes, icons, etc. on the interface state screenshot; the semantic features involve the text content on the interface state interface, such as button labels, error prompts, text in the input box, etc. The image processing layer can recognize and extract visual features and semantic features to generate structured information describing the interface content. For example, the information generation model recognizes the "network connection timeout" error prompt on the payment failure interface and uses it as part of the semantic feature.

[0058] In an embodiment of the present application, at least one interface content generated by the image processing layer based on the above-mentioned information generation model, at least one interface content and at least one target user operation data are respectively input into the text processing layer for word segmentation processing, and the word segmentation results are subjected to word form restoration processing to obtain a first word segmentation sequence and a second word segmentation sequence. Among them, the text processing layer is responsible for word segmentation processing and word form restoration processing of the at least one interface content and at least one target user operation data input as text. The word segmentation processing splits the text into individual words or phrases for subsequent feature extraction. The word form restoration processing restores the words to their basic form (such as the original form of the verb, the singular form of the noun, etc.) to ensure that the same word in different forms can be processed uniformly. For example, "click", "clicked", and "clicked" will all be restored to "click". After the above-mentioned word segmentation processing and word form restoration processing, the text processing layer will generate two word segmentation sequences: a first word segmentation sequence (corresponding to the interface content) and a second word segmentation sequence (corresponding to the target user operation data). The first word segmentation sequence and the second word segmentation sequence provide a structured text representation for subsequent feature extraction.

[0059] In an embodiment of the present application, the first segmentation sequence and the second segmentation sequence are generated by the text processing layer based on the above-mentioned information generation model, and the first segmentation sequence and the second segmentation sequence are respectively input into the feature extraction layer for extracting multi-dimensional features, and the multi-dimensional features include at least semantic features. Among them, the feature extraction layer is responsible for extracting multi-dimensional features from the segmentation sequence, and the multi-dimensional features are used for subsequent relevance calculations. The multi-dimensional features include at least semantic features, that is, the meaning and contextual relationship of the words. In addition, the feature extraction layer can also extract other types of features, such as grammatical features (syntactic roles of words), emotional features (emotional tendencies of texts), etc. Through natural language processing technology, the feature extraction layer can capture the deep semantic information in the first segmentation sequence and the second segmentation sequence, ensuring that the subsequent relevance calculation is more accurate. For example, the feature extraction layer can identify the semantic association between "payment failure" and "network connection timeout", and understand the similarity between these two words in different contexts.

[0060] In an embodiment of the present application, based on the multi-dimensional features of the first segmentation sequence and the multi-dimensional features of the second segmentation sequence obtained by the feature extraction layer of the above-mentioned information generation model, the multi-dimensional features of the first segmentation sequence and the multi-dimensional features of the second segmentation sequence are input into the relevance calculation layer for relevance calculation, so as to calculate the relevance between each interface content and the corresponding target user operation data. Among them, the relevance calculation layer is responsible for calculating the relevance between each interface content and the corresponding target user operation data. The calculation of the relevance is based on the multi-dimensional features of the two segmentation sequences, especially the semantic features, and the matching degree between the interface content and the user operation data is evaluated by comparing the keywords, phrases and semantic relationships in the interface content and the user operation data. For example, when a user encounters a payment failure problem when using a target application, and triggers a target event of an error report, the user operation data contains "payment failure", and the interface content displays an error prompt of "network connection timeout", the calculated relevance is 90, and the preset condition is that the relevance is greater than 85, then the interface state screenshot is selected as the target interface state screenshot that meets the set condition. The target interface status screenshot can not only clearly show the problems encountered by users, but also reflect the user's operation path, improve the accuracy of information feedback, and avoid the problem of low information feedback efficiency due to large data volume.

[0061] Furthermore, it should be noted that, in the case where the embodiments of the present application involve jumping between different application interfaces, the jumping methods involved in the embodiments of the present application include but are not limited to: jumping directly from an application interface to other application interfaces, jumping from a function module interface to a task interface first and then jumping to other application interfaces or function module interfaces after completing the corresponding task operation on the task interface; completing the corresponding task operation on the task interface includes but is not limited to: when the task interface is implemented as a game interface, completing the game operation on the game interface; when the task interface is implemented as an identity authentication interface, completing the identity authentication on the identity authentication interface; when the task interface is implemented as a recharge interface, completing the recharge operation on the recharge interface; and so on.

[0062] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0063] Figure 3 The following is a schematic diagram of an electronic device structure provided by an exemplary embodiment of the present application. Figure 3 As shown, the device includes: a memory 34 and a processor 35.

[0064] The memory 34 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device, user operation data sequences, user operation data, interface status screenshots, etc.

[0065] The memory 34 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0066] The processor 35 is coupled to the memory 34 and is used to execute the computer program in the memory 34, so as to be used for: an information processing method, comprising: in the process of running the target application, responding to multiple interactive operations of the user on at least one application interface, generating a user operation data sequence, the user operation data sequence including user operation data on the target user behavior path, at least one application interface is an application interface that appears on the target user behavior path; if a target event that needs to provide information feedback to the server occurs during the running of the target application, determining the start time and end time corresponding to the target event on the target user behavior path according to the information feedback scenario corresponding to the target event; obtaining the user operation data on the target user behavior path included in the user operation data sequence, and obtaining the user operation data between the start time and the end time. at least one target user operation data between the time periods; restoring at least one interactive operation of the user on the corresponding application interface based on the at least one target user operation data, and capturing at least one interface state screenshot corresponding to the at least one interactive operation during the restoration process; inputting the at least one target user operation data and the at least one interface state screenshot into an information generation model based on artificial intelligence AI, generating target text description information for information feedback based on the at least one target user operation data, and determining the target interface state screenshot for information feedback based on the at least one interface state screenshot and the at least one target user operation data; reporting the target text description information and the target interface state screenshot to the server, so that the server can perform information analysis on the target application based on the target text description information and the target interface state screenshot.

[0067] In an optional embodiment, the processor 35 pre-implants a stubbing code on the target user behavior path of the target application, and then generates a user operation data sequence in response to multiple interactive operations of the user on at least one application interface, including: sequentially running the stubbing code implanted on the target user behavior path, sensing the user's interactive operations on the corresponding application interface through the stubbing code, and recording detailed information of the interactive operations in sequence to obtain a user operation data sequence.

[0068] In an optional embodiment, if a target event that requires information feedback to the server occurs during the operation of the target application, the processor 35 determines the start time and end time corresponding to the target event on the target user behavior path according to the information feedback scenario corresponding to the target event, including: if a target event that requires information feedback to the server occurs during the operation of the target application, the information feedback scenario corresponding to the target event is determined to be a product improvement scenario or a functional abnormality scenario; if the information feedback scenario is a product improvement scenario and the target event is a user-triggered event, the occurrence time of the user-triggered event is obtained as the end time, and the target functional module to which the application interface where the user-triggered event belongs is determined, and the time of entering the target functional module is used as the start time; if the information feedback scenario is a functional abnormality scenario, the time when the target application exception occurs is obtained as the end time, and the target application interface where the exception occurs is determined, and the time of entering the target application interface is used as the start time.

[0069] In an optional embodiment, if a target event that requires information feedback to the server occurs during the operation of the target application, the processor 35 determines that the information feedback scenario corresponding to the target event is a product improvement scenario or a functional abnormality scenario, including: if the user initiates a first specified interactive operation during the operation of the target application, it is determined that the target event has occurred, and the target event is a user-triggered event, and the information feedback scenario corresponding to the target event is determined to be a product improvement scenario, and the first specified interactive operation represents the need for information feedback to the server; if an abnormality occurs during the operation of the target application, or the user initiates a second specified interactive operation after the target application is restarted due to an abnormality, it is determined that the target event has occurred, and the target event is an event in which an abnormality occurs in the target application, and the information feedback scenario corresponding to the target event is determined to be a functional abnormality scenario.

[0070] In an optional embodiment, the processor 35 restores at least one interactive operation of the user on the corresponding application interface based on at least one target user operation data, and captures at least one interface status screenshot corresponding to the at least one interactive operation during the restoration process, including: calling an automated recovery tool, setting environmental conditions according to the operating environment data contained in at least one target user operation data, and simulating the startup of the target application; after simulating the startup of the target application, traversing each target user operation data in the order between at least one target user operation data; for the currently traversed target user operation data, determining the application interface to which the currently traversed target user operation data belongs and the operation actions in the currently traversed target user operation data and their corresponding UI element identifiers, and performing the operation actions on the UI elements corresponding to the UI element identifiers on the application interface; and calling the screenshot API provided by the operating system to take a screenshot of the current application interface to obtain the interface status screenshot corresponding to the currently traversed target user operation data.

[0071] In an optional embodiment, after the processor 35 determines the application interface to which the currently traversed target user operation data belongs, it also includes: if the application interface to which the currently traversed target user operation data belongs is different from the application interface to which the previously traversed target user operation data belongs, then a page jump protocol is constructed, and based on the page jump protocol, the application interface to which the currently traversed target user operation data belongs is automatically jumped.

[0072] In an optional embodiment, the processor 35 inputs at least one target user operation data and at least one interface status screenshot into an information generation model based on artificial intelligence AI, generates target text description information for information feedback based on the at least one target user operation data, and determines the target interface status screenshot for information feedback based on the at least one interface status screenshot and the at least one target user operation data, including: inputting at least one target user operation data into the information generation model, generating target text description information for information feedback based on the at least one target user operation data; inputting at least one target user operation data and at least one interface status screenshot into the information generation model, and selecting a target interface status screenshot from at least one interface status screenshot based on the at least one interface status screenshot and the at least one target user operation data.

[0073] In an optional embodiment, the processor 35 inputs at least one target user operation data into the information generation model, and generates target text description information for information feedback based on the at least one target user operation data, including: inputting at least one target user operation data and an information feedback scenario into the information generation model, integrating key information in the at least one target user operation data to obtain initial text description information, the initial text description information describing the user's operation path and problem information; formatting and optimizing the initial text description information according to the text format and basic description information corresponding to the information feedback scenario to obtain intermediate text description information; adding classification labels adapted to the information feedback scenario to the intermediate text description information to obtain target text description information.

[0074] In an optional embodiment, the processor 35 inputs at least one target user operation data and at least one interface status screenshot into an information generation model, and selects a target interface status screenshot from the at least one interface status screenshot based on the at least one interface status screenshot and the at least one target user operation data, including: inputting at least one target user operation data and at least one interface status screenshot into an information generation model, extracting the interface content in the interface status screenshot corresponding to each target user operation data, and calculating the correlation between each interface content and the corresponding target user operation data; and selecting, according to the correlation between each interface content and the corresponding target user operation data, an interface status screenshot that meets the set correlation condition from at least one interface status screenshot as the target interface status screenshot.

[0075] In an optional embodiment, the information generation model includes: a text processing layer, a feature extraction layer, an image processing layer and a relevance calculation layer; the processor 35 inputs at least one target user operation data and at least one interface state screenshot into the information generation model, extracts the interface content in the interface state screenshot corresponding to each target user operation data, and calculates the relevance between each interface content and the corresponding target user operation data, including: inputting at least one interface state screenshot into the image processing layer, extracting the visual features and semantic features in at least one interface state screenshot, and generating at least one interface content according to the visual features and semantic features of at least one interface state screenshot; inputting at least one interface content and at least one target user operation data into the text processing layer for word segmentation processing, and performing word form restoration processing on the word segmentation results to obtain a first word segmentation sequence and a second word segmentation sequence; inputting the first word segmentation sequence and the second word segmentation sequence into the feature extraction layer for extracting multi-dimensional features, the multi-dimensional features at least including semantic features; inputting the multi-dimensional features of the first word segmentation sequence and the multi-dimensional features of the second word segmentation sequence into the relevance calculation layer for relevance calculation, and calculating the relevance between each interface content and the corresponding target user operation data.

[0076] Further, if Figure 3 As shown, the electronic device also includes: a communication component 36, a display 37, a power component 38, an audio component 39 and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 In addition, Figure 3 The components in the dotted box are optional components, not mandatory components, and the specific components depend on the product form of the working node.

[0077] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can implement the steps in the above method embodiment when the computer program is executed.

[0078] The embodiment of the present application also provides a computer program product, including: a computer program / instruction, when the computer program / instruction is executed by the processor, it enables the processor 35 to implement each step in the above method embodiment.

[0079] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0080] Above Figure 3 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0081] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0082] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0083] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0089] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0090] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0091] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0092] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An information processing method, characterized in that: include: During the operation of the target application, in response to multiple interactive operations of the user on at least one application interface, a user operation data sequence is generated, wherein the user operation data sequence includes user operation data on a target user behavior path, and the at least one application interface is an application interface that appears on the target user behavior path; If a target event that needs to provide information feedback to the server occurs during the operation of the target application, determine the start time and end time corresponding to the target event on the target user behavior path according to the information feedback scenario corresponding to the target event; Acquire at least one target user operation data between the start time and the end time from the user operation data on the target user behavior path included in the user operation data sequence; Restoring at least one interactive operation of the user on the corresponding application interface based on the at least one target user operation data, and capturing at least one interface state screenshot corresponding to the at least one interactive operation during the restoration process; Inputting the at least one target user operation data and the at least one interface state screenshot into an information generation model based on artificial intelligence AI, generating target text description information for information feedback based on the at least one target user operation data, and determining a target interface state screenshot for information feedback based on the at least one interface state screenshot and the at least one target user operation data; The target text description information and the target interface status screenshot are reported to the server, so that the server can perform information analysis on the target application based on the target text description information and the target interface status screenshot.

2. The method according to claim 1, characterized in that: The stubbing code is implanted in advance on the target user behavior path of the target application, and in response to multiple interactive operations of the user on at least one application interface, a user operation data sequence is generated, including: The stubbing codes implanted on the target user behavior path are run in sequence, the interactive operations of the user on the corresponding application interface are sensed through the stubbing codes, and detailed information of the interactive operations are recorded in sequence to obtain the user operation data sequence.

3. The method according to claim 1, characterized in that: If a target event that needs to provide information feedback to the server occurs during the operation of the target application, determining the start time and end time corresponding to the target event on the target user behavior path according to the information feedback scenario corresponding to the target event includes: If a target event that requires information feedback to the server occurs during the operation of the target application, determining that the information feedback scenario corresponding to the target event is a product improvement scenario or a function abnormality scenario; If the information feedback scenario is a product improvement scenario and the target event is a user-triggered event, the occurrence time of the user-triggered event is obtained as the end time, and the target function module to which the application interface where the user-triggered event is located is determined, and the time of entering the target function module is taken as the start time; If the information feedback scenario is a functional abnormality scenario, the time when the abnormality occurs in the target application is obtained as the end time, and the target application interface where the abnormality occurs is determined, and the time of entering the target application interface is used as the start time.

4. The method according to claim 3, characterized in that If a target event that requires information feedback to the server occurs during the operation of the target application, determining that the information feedback scenario corresponding to the target event is a product improvement scenario or a function abnormality scenario includes: If the user initiates a first designated interactive operation during the operation of the target application, it is determined that the target event occurs, and the target event is a user-triggered event, and it is determined that the information feedback scenario corresponding to the target event is a product improvement scenario, the first designated interactive operation indicates that information feedback needs to be provided to the server; If an exception occurs during the operation of the target application, or the user initiates a second designated interactive operation after the target application is restarted due to an exception, it is determined that the target event has occurred, and the target event is an event in which an exception occurs in the target application, and the information feedback scenario corresponding to the target event is determined to be a functional abnormality scenario.

5. The method according to claim 1, characterized in that: Restoring at least one interactive operation of a user on a corresponding application interface based on the at least one target user operation data, and capturing at least one interface state screenshot corresponding to the at least one interactive operation during the restoration process, including: Invoking an automated recovery tool, setting environmental conditions according to the operating environment data included in the at least one target user operation data, and simulating starting the target application; After simulating the start of the target application, traversing each target user operation data in the order between the at least one target user operation data; For the currently traversed target user operation data, determine the application interface to which the currently traversed target user operation data belongs, the operation action in the currently traversed target user operation data and its corresponding UI element identifier, and execute the operation action on the UI element corresponding to the UI element identifier on the application interface; and call the screenshot API provided by the operating system to take a screenshot of the current application interface to obtain a screenshot of the interface status corresponding to the currently traversed target user operation data.

6. The method according to claim 5, characterized in that After determining the application interface to which the currently traversed target user operation data belongs, it also includes: If the application interface to which the currently traversed target user operation data belongs is different from the application interface to which the previously traversed target user operation data belongs, a page jump protocol is constructed, and based on the page jump protocol, the page automatically jumps to the application interface to which the currently traversed target user operation data belongs.

7. The method according to any one of claims 1 to 6, characterized in that: Inputting the at least one target user operation data and the at least one interface state screenshot into an information generation model based on artificial intelligence AI, generating target text description information for information feedback based on the at least one target user operation data, and determining a target interface state screenshot for information feedback based on the at least one interface state screenshot and the at least one target user operation data, including: Inputting the at least one target user operation data into the information generation model, and generating target text description information for information feedback based on the at least one target user operation data; The at least one target user operation data and the at least one interface state screenshot are input into the information generation model, and based on the at least one interface state screenshot and the at least one target user operation data, the target interface state screenshot is selected from the at least one interface state screenshot.

8. The method according to claim 7, characterized in that Inputting the at least one target user operation data into the information generation model, and generating target text description information for information feedback based on the at least one target user operation data, including: Inputting the at least one target user operation data and the information feedback scenario into the information generation model, integrating key information in the at least one target user operation data to obtain initial text description information, wherein the initial text description information describes the user's operation path and problem information; According to the text format and basic description information corresponding to the information feedback scenario, format correction and content optimization are performed on the initial text description information to obtain intermediate text description information; A classification label adapted to the information feedback scenario is added to the intermediate text description information to obtain the target text description information.

9. The method according to claim 7, characterized in that: Inputting the at least one target user operation data and the at least one interface state screenshot into the information generation model, and selecting the target interface state screenshot from the at least one interface state screenshot based on the at least one interface state screenshot and the at least one target user operation data, comprising: Inputting the at least one target user operation data and the at least one interface state screenshot into the information generation model, extracting the interface content in the interface state screenshot corresponding to each target user operation data, and calculating the correlation between each interface content and the corresponding target user operation data; According to the relevance between each interface content and the corresponding target user operation data, an interface state screenshot that meets a set relevance condition is selected from at least one interface state screenshot as the target interface state screenshot.

10. The method according to claim 9, characterized in that: The information generation model includes: a text processing layer, a feature extraction layer, an image processing layer and a relevance calculation layer; Inputting the at least one target user operation data and the at least one interface state screenshot into the information generation model, extracting the interface content in the interface state screenshot corresponding to each target user operation data, and calculating the correlation between each interface content and the corresponding target user operation data, including: Inputting the at least one interface state screenshot into an image processing layer, extracting visual features and semantic features in the at least one interface state screenshot, and generating at least one interface content according to the visual features and semantic features of the at least one interface state screenshot; Inputting the at least one interface content and the at least one target user operation data into the text processing layer for word segmentation processing respectively, and performing word form restoration processing on the word segmentation results to obtain a first word segmentation sequence and a second word segmentation sequence; Inputting the first word segmentation sequence and the second word segmentation sequence into a feature extraction layer to extract multi-dimensional features, wherein the multi-dimensional features at least include semantic features; The multi-dimensional features of the first word segmentation sequence and the multi-dimensional features of the second word segmentation sequence are input into the relevance calculation layer to perform relevance calculation, based on the relevance between each interface content and the corresponding target user operation data.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps of the method according to any one of claims 1 to 10.

Images