User operation display method and device, electronic equipment and storage medium

Abstract

The invention provides a user operation display method and device, electronic equipment and a storage medium, and relates to the technical field of computers. In the application, in response to at least one operation of a target user for a target operation platform, interface response information corresponding to the target operation platform based on the at least one operation is recorded; wherein each piece of interface response information can be used for indicating the service provided by the target operation platform for the target user according to the corresponding operation; and then, on the basis of the at least one piece of interface response information and the interface touch information associated with the at least one piece of operation, generating an operation track video of a target user for displaying the at least one piece of operation. Therefore, a burying point does not need to be set for the operation of the target user, and the user operation can be automatically captured; moreover, the operation track of the target user for the target operation platform is visually displayed by generating the operation track video, namely, the operation track of the target user is better displayed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for displaying user operations. Background Art

[0002] In various operating platforms (eg, Android platform, etc.), the operation trajectory of users (ie, objects) in the operating platforms plays a vital role in analyzing user behaviors.

[0003] At present, since the tracking point (or event tracking) method is usually adopted, each operation of the user is recorded separately and reported to the tracking point platform. Therefore, when checking the operation track of the user, the tracking point information of the user can be queried through the tracking point platform, so as to obtain the operation track of the user in text form.

[0004] However, when using the above-mentioned operation track display method, it is necessary to set a tracking point for each operation of the user, which may result in a lot of time to complete the setting of the tracking point, that is, the setting operation of the tracking point is relatively cumbersome; and, displaying the user's operation track in text form cannot well display the user's operation track. Summary of the invention

[0005] The embodiments of the present application provide a method, device, electronic device and storage medium for displaying user operations, so as to improve the problem in the prior art that the setting operation of the tracking point is relatively cumbersome and the text form cannot well display the user's operation trajectory.

[0006] In a first aspect, an embodiment of the present application provides a method for displaying user operations, the method comprising:

[0007] In response to at least one operation of the target user on the target operating platform, record the interface response information corresponding to the target operating platform based on the at least one operation; wherein each interface response information is used to indicate the service provided by the target operating platform for the target user according to the corresponding operation;

[0008] Based on at least one interface response information and at least one interface touch information respectively associated with the operation, an operation track video of the target user is generated; wherein the operation track video is used to display the at least one operation.

[0009] In an optional embodiment, recording the interface response information corresponding to the target operating platform based on at least one operation includes:

[0010] Determining at least one interface information recording method based on the operation type corresponding to the at least one operation;

[0011] Based on at least one interface information recording method, interface response information corresponding to at least one operation of the target operating platform is recorded.

[0012] In an optional embodiment, generating an operation track video of a target user based on at least one interface response information and at least one interface touch information respectively associated with the operation includes:

[0013] Obtaining at least one root view corresponding to each operation based on at least one interface response information; wherein each root view is used to indicate the interface framework layout of the target operating platform under the corresponding operation;

[0014] Based on the interface touch information and the subview set respectively corresponding to at least one root view, draw an interface image respectively corresponding to at least one operation;

[0015] Based on the at least one interface image obtained, an operation trajectory video of the target user is generated.

[0016] In an optional embodiment, based on the interface touch information and the subview set respectively corresponding to at least one root view, drawing an interface image respectively corresponding to at least one operation includes:

[0017] For at least one operation, perform the following operations:

[0018] Based on a root view corresponding to an operation, determining an interface object corresponding to an operation; wherein the interface object is used to indicate an operation window corresponding to an operation in a target operation platform;

[0019] Adopting the touch information acquisition method set for the interface object to obtain the corresponding interface touch information;

[0020] Draw an operation interface image based on the interface touch information and the subview collection of the root view.

[0021] In an optional embodiment, drawing an interface image of an operation based on the interface touch information and the subview set of the root view includes:

[0022] Traversing the sub-view set based on at least one sub-view response included in the interface response information to obtain at least one sub-view corresponding to an operation;

[0023] Based on the interface touch information and at least one subview, an interface image of an operation is drawn.

[0024] In an optional embodiment, after drawing an interface image of an operation based on the interface touch information and the subview set of the root view, the method further includes:

[0025] Encoding video data of an operation interface image to obtain a corresponding video data stream;

[0026] Based on the obtained at least one interface image, generating an operation track video of the target user includes:

[0027] Based on the video data streams respectively corresponding to at least one interface image, an operation track video of the target user is generated.

[0028] In an optional embodiment, after generating the operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation, the method further includes:

[0029] In response to an operation track query request for a target user, an operation track video of the target user is screened out from a preset operation track video set; wherein the operation track video set includes: an operation track video corresponding to at least one user;

[0030] Play the target user's operation trajectory video.

[0031] In a second aspect, an embodiment of the present application further provides a user-operated display device, the device comprising:

[0032] An information recording module, configured to record, in response to at least one operation of a target user on a target operating platform, interface response information corresponding to the target operating platform based on the at least one operation; wherein each interface response information is used to indicate a service provided by the target operating platform for the target user according to the corresponding operation;

[0033] The video generation module is used to generate an operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation; wherein the operation track video is used to display at least one operation.

[0034] In an optional embodiment, when recording the interface response information corresponding to the target operating platform based on at least one operation, the information recording module is specifically used to:

[0035] Determining at least one interface information recording method based on the operation type corresponding to the at least one operation;

[0036] Based on at least one interface information recording method, interface response information corresponding to at least one operation of the target operating platform is recorded.

[0037] In an optional embodiment, when generating the operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation, the video generation module is specifically used to:

[0038] Obtaining at least one root view corresponding to each operation based on at least one interface response information; wherein each root view is used to indicate the interface framework layout of the target operating platform under the corresponding operation;

[0039] Based on the interface touch information and the subview set respectively corresponding to at least one root view, draw an interface image respectively corresponding to at least one operation;

[0040] Based on the at least one interface image obtained, an operation trajectory video of the target user is generated.

[0041] In an optional embodiment, when drawing an interface image corresponding to at least one operation based on the interface touch information and the subview set corresponding to at least one root view, the video generation module is specifically used to:

[0042] For at least one operation, perform the following operations:

[0043] Based on a root view corresponding to an operation, determining an interface object corresponding to an operation; wherein the interface object is used to indicate an operation window corresponding to an operation in a target operation platform;

[0044] Adopting the touch information acquisition method set for the interface object to obtain the corresponding interface touch information;

[0045] Draw an operation interface image based on the interface touch information and the subview collection of the root view.

[0046] In an optional embodiment, when drawing an interface image of an operation based on the interface touch information and the subview set of the root view, the video generation module is specifically used to:

[0047] Traversing the sub-view set based on at least one sub-view response included in the interface response information to obtain at least one sub-view corresponding to an operation;

[0048] Based on the interface touch information and at least one subview, an interface image of an operation is drawn.

[0049] In an optional embodiment, after drawing an interface image of an operation based on the interface touch information and the subview set of the root view, the video generation module is further used to:

[0050] Encoding video data of an operation interface image to obtain a corresponding video data stream;

[0051] When generating the operation track video of the target user based on the at least one interface image obtained, the video generation module is specifically used to:

[0052] Based on the video data streams respectively corresponding to at least one interface image, an operation track video of the target user is generated.

[0053] In an optional embodiment, after generating the target user's operation trajectory video based on at least one interface response information and at least one interface touch information respectively associated with the operation, the device further includes an operation display module, which is specifically used to:

[0054] In response to an operation track query request for a target user, an operation track video of the target user is screened out from a preset operation track video set; wherein the operation track video set includes: an operation track video corresponding to at least one user;

[0055] Play the target user's operation trajectory video.

[0056] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0057] Processor; and

[0058] Memory for storing programs,

[0059] The program includes instructions, which, when executed by a processor, cause the processor to execute the method for displaying user operations as described in the first aspect.

[0060] In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method for displaying user operations as described in the first aspect.

[0061] In a fifth aspect, the present application provides a computer program product, which, when called by a computer, enables the computer to execute the steps of the method for displaying user operations as described in the first aspect.

[0062] The beneficial effects of this application are as follows:

[0063] In the method for displaying user operations provided in the embodiment of the present application, in response to at least one operation of the target user on the target operating platform, the interface response information corresponding to the target operating platform based on the at least one operation is recorded; wherein each interface response information can be used to indicate the service provided by the target operating platform to the target user according to the corresponding operation; then, based on the at least one interface response information and the interface touch information respectively associated with the at least one operation, an operation track video of the target user for displaying the aforementioned at least one operation can be generated. It can be seen that there is no need to set a tracking point for the target user's operation, and the user operation can be automatically captured; and by generating the operation track video, the operation track of the target user on the target operating platform is visually displayed, that is, the operation track of the target user is better displayed.

[0064] In addition, other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described here are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0066] Figure 1 A schematic diagram of an optional system architecture applicable to the embodiments of the present application;

[0067] Figure 2 A schematic diagram of an implementation flow of a method for displaying user operations provided in an embodiment of the present application;

[0068] Figure 3 A logical schematic diagram of recording interface response information provided by an embodiment of the present application;

[0069] Figure 4 A schematic diagram of event distribution logic provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of an implementation flow of an interface image drawing method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a specific scenario for obtaining interface touch information provided in an embodiment of the present application;

[0072] Figure 7 A logical schematic diagram for obtaining view display information provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of the structure of a user-operated display device provided in an embodiment of the present application;

[0074] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0076] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0077] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0078] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0079] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0080] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0081] 1. Tracking: refers to the technology and implementation process of capturing, processing and sending specific user behaviors or events, which can be used to collect user behavior data. For example, if you want to know which buttons a user clicked, which pages he viewed, what he did, etc. in an application (APP), you can do it through tracking.

[0082] 2. Reflection: It is a mechanism provided by the Java language that allows the program to dynamically obtain and manipulate the internal information of any class at runtime. Exemplarily, the aforementioned internal information may include the attributes, methods, constructors, etc. of the class.

[0083] 3. Hook: It is a special message processing mechanism that can monitor various event messages in the system or process, intercept messages sent to the target window and process them. There are many types of hooks, and each hook can intercept corresponding messages, such as keyboard hook can intercept keyboard messages, shell hook can intercept, start and close APP messages, etc.

[0084] Hooks can be divided into thread hooks and system hooks. Thread hooks can monitor event messages of a specified thread, while system hooks monitor event messages of all threads in the system. And because system hooks affect all apps in the system, hook functions must usually be placed in independent dynamic link libraries (DLLs). Therefore, hooks are actually message interception mechanisms that can intercept messages of a single process (i.e., thread hooks), messages of all processes (i.e., system hooks), and can also perform custom processing on intercepted messages.

[0085] 4. ViewGroup: It is a combination or collection of a group of views, which may contain multiple sub-views. When the user touches or touches the screen, the touch area may be within the display range of the ViewGroup or on its sub-views.

[0086] 5. A bitmap, also known as a dot matrix image or a raster image, is composed of individual points called pixels (picture elements). These points can be arranged and dyed differently to form patterns, which can show color changes and subtle color transitions, producing realistic effects.

[0087] Based on the above-mentioned nouns and related terminology explanations, the design concept of the embodiments of the present application is briefly introduced below:

[0088] When performing user behavior analysis on various operating platforms, it is usually necessary to view the operation track of the user (or object) on the corresponding operating platform. For example, the operation track of one or more apps deployed by the user on the Android platform. At present, the main method is to record each user's operation separately and report it to the tracking point platform, query the user's tracking point information through the tracking point platform, and obtain the user's operation track in text form.

[0089] However, using the above method, any operation of the user on the operating platform needs to set a corresponding tracking point. Therefore, when the types of user operations are large, the setting of tracking points is cumbersome; and the user's operation trajectory can only be viewed in text form, which cannot better display the user's operation trajectory.

[0090] In view of this, in order to solve the above problems, the embodiment of the present application proposes a method for displaying user operations, which specifically includes: in response to at least one operation of the target user on the target operation platform, recording the interface response information corresponding to the target operation platform based on the at least one operation; wherein each interface response information is used to indicate the service provided by the target operation platform to the target user according to the corresponding operation; then, based on the at least one interface response information and the interface touch information respectively associated with the at least one operation, the operation track video of the target user can be generated; wherein the operation track video is used to display at least one operation. In this way, according to recording the interface response information corresponding to at least one operation of the target user on the target operation platform and the interface touch information respectively associated with the at least one operation, the operation track video of the target user is generated, and there is no need to set a tracking point for each operation of the target user; and the operation track of the target user is displayed by the operation track video, so as to realize the visualization of the operation track of the target user. Therefore, the problem that the setting operation of the tracking point is cumbersome and the text form cannot well display the user's operation track in the prior art is improved.

[0091] In particular, the preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0092] See also Figure 1 As shown, it is a schematic diagram of a system architecture applicable to an embodiment of the present application, and the system architecture may include: a target terminal 101 and a server 102. The target terminal 101 and the server 102 may exchange information through a communication network, wherein the communication mode adopted by the communication network may include: a wireless communication mode and a wired communication mode.

[0093] Exemplarily, the target terminal 101 can access the network through cellular mobile communication technology to communicate with the server 102. The cellular mobile communication technology, for example, includes the fifth generation mobile communication (5th generation mobile networks, 5G) technology or the next generation mobile communication technology.

[0094] Optionally, the target terminal 101 may access the network through a short-range wireless communication method to communicate with the server 102. The short-range wireless communication method, for example, includes wireless fidelity (Wi-Fi) technology.

[0095] The embodiment of the present application does not impose any restriction on the number of communication devices involved in the above system architecture. For example, the above system architecture may include more target terminals or other network devices. Figure 1 As shown, only the target terminal 101 and the server 102 are described as examples, and the above-mentioned communication devices and their respective functions are briefly introduced below.

[0096] The target terminal 101 is a device that can provide voice and / or data connectivity to the user, and can be a device that supports wired and / or wireless connection.

[0097] Exemplarily, the target terminal 101 may include, but is not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminal devices in industrial control, wireless terminal devices in unmanned driving, wireless terminal devices in smart grids, wireless terminal devices in transportation safety, wireless terminal devices in smart cities, or wireless terminal devices in smart homes, etc.

[0098] In addition, the target terminal 101 may be installed with a relevant client, which may be software, such as an APP, a browser, a short video software, etc., or a web page, a small program, etc. It should be noted that in the embodiment of the present application, the target terminal 101 may enable the above-mentioned client related to the display of user operations to send the interface response information corresponding to at least one operation of the target user on the target operating platform, and / or the interface touch information respectively associated with the aforementioned at least one operation, etc., to the server 102, so as to perform subsequent method steps such as display of user operations.

[0099] Optionally, the target terminal 101 may enable the above-mentioned client related to the display of the user operation to send an operation track query request for the target user to the server 102, so as to play / display the operation track video of the target user.

[0100] In the embodiment of the present application, the type of the operating platform of the target terminal 101 (ie, the target operating platform) is not limited. Figure 1 As shown, the aforementioned target operating platform can be any one of the operating systems (OS) such as Android (Google), iOS (Apple), Windows phone (Microsoft), etc. Of course, the aforementioned target operating platform can also be other operating platforms, such as Symbian (Nokia), BlackBerry OS (BlackBerry) or Harmony (Hongmeng).

[0101] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0102] It is worth mentioning that in an embodiment of the present application, the server 102 can record the interface response information corresponding to the target operating platform based on the at least one operation in response to the target user; wherein each interface response information can be used to indicate the service provided by the target operating platform to the target user according to the corresponding operation; finally, based on the at least one interface response information and the interface touch information respectively associated with the at least one operation, an operation trajectory video of the target user is generated; wherein the operation trajectory video is used to display at least one operation.

[0103] Optionally, in order to avoid synthesizing the operation track video of the target user on the server 102 (i.e., the service end), it may cause the problem of large back-end resource consumption and computing burden. The target terminal 101 can also perform the display method of the user operation provided by the embodiment of the present application. In addition, since the video compression technology can remove redundant information and use coding technology to represent the image data in a more compact form, after the target terminal 101 synthesizes the operation track video of the target user, the operation track video of the aforementioned target user is uploaded to the server 102, which can also reduce the size of the uploaded file.

[0104] The following describes the method for displaying user operations provided by an exemplary embodiment of the present application in combination with the above-mentioned system architecture and with reference to the accompanying drawings. It should be noted that the above-mentioned system architecture is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0105] See also Figure 2 As shown, it is a schematic diagram of an implementation flow of a user operation display method provided in an embodiment of the present application. For the convenience of description and understanding, the execution subject takes a server as an example. The specific implementation flow of the method is as follows:

[0106] S201: In response to at least one operation of a target user on a target operating platform, record interface response information of the target operating platform corresponding to the at least one operation.

[0107] Each interface response information can be used to indicate the service provided by the target operating platform to the target user according to the corresponding operation. The aforementioned service provided to the target user can be a network service, a communication service, or other services, which are not specifically limited in the embodiments of the present application.

[0108] In addition, since the target user's operations on the target operating platform usually cause interface changes, each of the aforementioned interface response information can also be used to indicate the initial interface of the target operating platform before the corresponding operation and the response interface of the target operating platform after the corresponding operation, that is, it can indicate the interface change information corresponding to the corresponding operation.

[0109] It should be noted that the at least one operation mentioned above may be multiple operations performed by the target user on the target operating platform within different time ranges, and of course, may also be continuous or intermittent operations performed by the target user within the same time range.

[0110] In an alternative implementation, see Figure 3 As shown, when executing step S201, the server can determine at least one interface information recording method based on the operation type corresponding to the at least one operation, and then record the interface response information corresponding to the at least one operation of the target operation platform based on the at least one interface information recording method. Exemplarily, the operation type can include but is not limited to gesture operation and input operation.

[0111] Based on the above method, the interface information recording method set according to different operation types can record the interface changes caused by the target user's operation as comprehensively as possible to ensure that the user's operation trajectory can be better displayed later.

[0112] Take the target operating platform as Android platform as an example, refer to Figure 4 As shown in the figure, the user's gesture operation is usually first distributed to the dispatchTouchEvent() of Window.Callback, and then distributed to each child View through the root view (DecorView) and the View tree level by level to consume (or process) the event (i.e. onTouchEvent). Among them, the events (Touch Event) in the Android platform mainly include tap, long press, drag, slide and cancel operations. All events can be composed of three parts: press (ACTION_DOWN), move (ACTTION_MOVE) and lift (ACTION_UP).

[0113] Therefore, if the target user's operation is a gesture operation, the default Window.Callback() can be overwritten with a custom Window.Callback() proxy to rewrite dispatchTouchEvent(). Furthermore, when a gesture operation of the target user is detected in dispatchTouchEvent(), the gesture operation of the target user can be monitored through the custom Window.Callback(), thereby recording the interface response information of the target operation platform based on the aforementioned gesture operation.

[0114] For another example, if the target user's operation is an input operation, the target user's input operation can also be monitored according to editText.addTextChangedListener(), so as to record the interface response information of the target operation platform based on the aforementioned input operation.

[0115] Optionally, in order to more comprehensively and carefully record the changes in the interface (or page) information in the target operating platform, you can also monitor the interface layout changes of the target operating platform through decorView.getViewTreeObserver().addOnGlobalLayoutListener() to obtain more comprehensive interface response information.

[0116] S202: Generate an operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation.

[0117] The operation track video is used to show at least one operation of the target user on the target operating platform, that is, the operation track video can show the operation track of the target user on the target operating platform. Optionally, the interface touch information is used to indicate the touch position of the target user on the target operating platform. In other words, the interface click position of the target user on the target operating platform can be determined based on the interface touch information.

[0118] In an optional implementation, when executing step S202, after obtaining the interface response information corresponding to the above-mentioned at least one operation, the server can obtain the root view corresponding to the at least one operation based on the at least one interface response information, thereby drawing the interface image corresponding to the at least one operation based on the interface touch information and the subview set corresponding to the at least one root view, and then generating the operation track video of the target user based on the obtained at least one interface image. Among them, each root view can be used to indicate the interface framework layout of the target operation platform under the corresponding operation. In other words, the root view is essentially a frame layout (FrameLayout). The aforementioned subview set can include multiple Views other than the root view (or root View) in the View tree. In this way, since the subview set will display the actual display interface of the target operation platform, the interface image corresponding to the at least one operation drawn according to the interface touch information corresponding to the at least one root view and the subview set can visually reflect the interface changes of the target operation platform under at least one operation of the target user, so as to better show the operation track of the target user.

[0119] Taking the interface response information corresponding to five operations as an example, after obtaining the five interface response information, the server can obtain the root views corresponding to the five operations according to the five interface response information. Then, the interface images corresponding to the five operations can be drawn according to the interface touch information and subview sets corresponding to the five root views. Since the operations of the target user usually trigger changes in the interface, the five operations correspond to at least two interface images (for example, an initial interface image and a response interface image). Based on the interface images corresponding to the five operations (for example, 10 interface images), the server can generate an operation trajectory video of the target user according to the 10 interface images.

[0120] In order to better display the operation trajectory of the target user, the display time of each interface image and the connection method and / or time interval between adjacent interface images can also be flexibly set, which is not limited in the embodiments of the present application.

[0121] In an optional implementation, when the server draws the interface image corresponding to at least one operation based on the interface touch information and the subview set corresponding to at least one root view, for any one of the aforementioned at least one operation, such as the first operation, refer to Figure 5 As shown, you can perform the following operations respectively:

[0122] S501: Determine an interface object corresponding to the first operation based on a root view corresponding to the first operation.

[0123] The interface object is used to indicate the operation window corresponding to the first operation in the target operation platform. Still taking the target operation platform as the Android platform as an example, the aforementioned interface object (i.e., MotionEvent) can be a Window object, which can represent a window opened in a browser, and is the top-level object and core object of the browser object model (browser object model, BOM). Other objects of the BOM are all sub-objects (or attributes) of the Window object.

[0124] Optionally, the above sub-objects may include Document object, i.e., document object; Location object, representing the current universal resource locator (URL) information of the browser; Navigator object, representing the browser information itself; Screen object, representing the client screen information; History object, representing the browser access history information.

[0125] For example, when executing step S501, since the interface seen by the user in the Android platform is mainly displayed in the form of Activity or Dialog (i.e., page or pop-up window), and when the Activity or Dialog is displayed, the View added to the WindowManager is the root view that contains the Window object, the Window object can be obtained based on the root view through reflection.

[0126] It should be noted that the above-mentioned interface object may also have other names, which is not limited in the embodiments of the present application.

[0127] S502: Acquire corresponding interface touch information using a touch information acquisition method set for the interface object.

[0128] The above interface touch information can be the position coordinates of the touch point of the target user on the screen or interface (i.e. the click position of the target user). Taking the Android platform as an example, there are usually two coordinate systems in the Android platform: the Android coordinate system and the view coordinate system. Figure 6 As shown in (a) in the figure, the Android coordinate system refers to the top left corner of the screen as the origin of the Android coordinate system, the right is the positive direction of the X axis, and the downward is the positive direction of the Y axis. At this time, the above touch information acquisition method can be getRawX() and getRawY(). getRawX() is used to obtain the distance from the touch point to the left side of the screen, that is, the absolute X coordinate; getRawY() is used to obtain the distance from the touch point to the top edge of the screen, that is, the absolute Y coordinate.

[0129] See also Figure 6As shown in (b) in the figure, the view coordinate system refers to the top left corner of the control (such as the View in the ViewGroup) as the origin of the view coordinate system, with the right as the positive direction of the X axis and the bottom as the positive direction of the Y axis. At this time, the above touch information acquisition method can be getX() and getY(). getX() is used to obtain the distance from the touch point to the left side of the control, that is, the view X coordinate; getY() is used to obtain the distance from the touch point to the top edge of the control, that is, the view Y coordinate. Optional, such as Figure 6 As shown in (b), you can also use getTop() to get the distance from the top of the child control (View) to the top of the parent control (ViewGroup), getLeft() to get the distance from the left of the child control to the left of the parent control, getRight() to get the distance from the right of the child control to the left of the parent control, and getBottom() to get the distance from the bottom of the child control to the top of the parent control.

[0130] S503: Drawing an interface image of the first operation based on the interface touch information and the subview set of the root view.

[0131] In an optional implementation, when executing step S503, the server may traverse the subview set corresponding to the first operation based on at least one subview response included in the interface response information, obtain at least one subview corresponding to the first operation, and then draw the interface image of the first operation based on the interface touch information and the at least one subview. The aforementioned subview response may be mFirstTouchTarget, child attributes, etc. that record the response to the user operation.

[0132] Exemplarily, the server can obtain the root view through the interface response information (e.g., Window) recorded by the aforementioned custom Window.Callback, and then traverse from the root view in sequence through reflection to obtain the mFirstTouchTarget and child attributes that record the response to the user operation, and then obtain at least one subview that responds to the event (i.e., the target user's operation) from the subview set corresponding to the root view. In this way, the aforementioned at least one subview can be drawn to the bitmap through the canvas according to the drawing draw() method of the aforementioned at least one subview and the corresponding interface touch information (e.g., the click position of the touch point obtained according to getRawX() and getRawY()), thereby obtaining the interface image of the corresponding operation.

[0133] It should be noted that all views on the Android platform are displayed through WindowManager's addView(). For example, see Figure 7As shown, the implementation of WindowManager is actually WindowManagerImpl(), in which addView() of WindowManagerImpl() calls addView() of WindowManagerGlobal(), and addView() of WindowManagerGlobal() adds View to the ArrayList mViews. <view>In the collection (that is, the root view collection), you can monitor the display of all root views by hooking this mViews through reflection.

[0134] It can be seen that based on the method steps recorded in the above S501 to S503, the interface image drawn includes both the page view information during the target user's operation and the click position information of the target user. Therefore, it can more comprehensively reflect at least one operation (i.e., the operation trajectory) of the target user on the target operation platform, and also realize the conversion of interface information into image data.

[0135] In an optional implementation, after executing step S503, the server may also encode the video data of the interface image of the first operation to obtain a corresponding video data stream; in this way, the operation trajectory video of the target user may be subsequently generated based on the video data stream corresponding to at least one of the interface images.

[0136] For example, the server can use the codec component of the Android platform (e.g., MediaCodec) to obtain the binary data (i.e., video stream data) of each interface image in real time, thereby writing the binary data into the target user's operation track video in real time. In this way, there is no need to store the obtained interface images locally first, and then synthesize the target user's operation track video based on the obtained multiple interface images when the interface image generation operation is stopped, thereby avoiding the problem of a long time spent on synthesizing multiple interface images into a video.

[0137] Furthermore, after generating the operation track video of the target user, the server may also upload the operation track video of the target user to the track tracking platform so as to subsequently view the operation track of the target user.

[0138] In an optional implementation, after executing step S202, the server may also respond to the operation track query request for the target user, and filter out the operation track video of the target user from the preset operation track video set; and play the operation track video of the target user. The aforementioned operation track video set may include: an operation track video corresponding to at least one user. It can be seen that the operation track of the aforementioned target user is played in the form of video instead of being displayed in the form of text, which can better show the operation track of the target user, that is, realize the visual presentation of the operation track.

[0139] Exemplarily, the server can obtain the user identifier of the target user from the above-mentioned operation trajectory query request; then, it can filter out the target operation trajectory video that matches the user identifier from the preset operation trajectory video set, and use the target operation trajectory video as the operation trajectory video of the target user; finally, it can play the operation trajectory video of the target user.

[0140] In summary, in the method for displaying user operations provided in the embodiment of the present application, in response to at least one operation of the target user on the target operation platform, the interface response information corresponding to the target operation platform based on the at least one operation is recorded; wherein, each interface response information can be used to indicate the service provided by the target operation platform to the target user according to the corresponding operation; then, based on the at least one interface response information and the interface touch information respectively associated with the at least one operation, an operation track video of the target user for displaying the aforementioned at least one operation can be generated. It can be seen that there is no need to set a tracking point for the operation of the target user, and the user operation can be automatically captured; and by generating an operation track video, the operation track of the target user on the target operation platform is visually displayed, that is, the operation track of the target user is better displayed. In addition, since the embodiment of the present application does not need to set a tracking point, then when viewing the operation track of the target object later, there is no need to view the tracking point document to know the timing, meaning and other information of the tracking point triggering, that is, there is no need to apply for permission to view the operation track.

[0141] Further, based on the same technical concept, the embodiment of the present application provides a user-operated display device, which is used to implement the above method flow of the embodiment of the present application. Figure 8 As shown, the user-operated display device 800 includes: an information recording module 801 and a video generation module 802, wherein:

[0142] The information recording module 801 is used to record the interface response information corresponding to the target operating platform based on the at least one operation of the target user on the target operating platform; wherein each interface response information is used to indicate the service provided by the target operating platform for the target user according to the corresponding operation;

[0143] The video generation module 802 is used to generate an operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation; wherein the operation track video is used to display at least one operation.

[0144] In an optional embodiment, when recording the interface response information corresponding to the target operating platform based on at least one operation, the information recording module 801 is specifically used to:

[0145] Determining at least one interface information recording method based on the operation type corresponding to the at least one operation;

[0146] Based on at least one interface information recording method, interface response information corresponding to at least one operation of the target operating platform is recorded.

[0147] In an optional embodiment, when generating the operation track video of the target user based on at least one interface response information and at least one interface touch information respectively associated with the operation, the video generation module 802 is specifically used to:

[0148] Obtaining at least one root view corresponding to each operation based on at least one interface response information; wherein each root view is used to indicate the interface framework layout of the target operating platform under the corresponding operation;

[0149] Based on the interface touch information and the subview set respectively corresponding to at least one root view, draw an interface image respectively corresponding to at least one operation;

[0150] Based on the at least one interface image obtained, an operation trajectory video of the target user is generated.

[0151] In an optional embodiment, when drawing an interface image corresponding to at least one operation based on the interface touch information and the subview set corresponding to at least one root view, the video generation module 802 is specifically used to:

[0152] For at least one operation, perform the following operations:

[0153] Based on a root view corresponding to an operation, determining an interface object corresponding to an operation; wherein the interface object is used to indicate an operation window corresponding to an operation in a target operation platform;

[0154] Adopting the touch information acquisition method set for the interface object to obtain the corresponding interface touch information;

[0155] Draw an operation interface image based on the interface touch information and the subview collection of the root view.

[0156] In an optional embodiment, when drawing an interface image of an operation based on the interface touch information and the subview set of the root view, the video generation module 802 is specifically used to:

[0157] Traversing the sub-view set based on at least one sub-view response included in the interface response information to obtain at least one sub-view corresponding to an operation;

[0158] Based on the interface touch information and at least one subview, an interface image of an operation is drawn.

[0159] In an optional embodiment, after drawing an interface image of an operation based on the interface touch information and the subview set of the root view, the video generation module 802 is further used to:

[0160] Encoding video data of an operation interface image to obtain a corresponding video data stream;

[0161] When generating the operation track video of the target user based on the obtained at least one interface image, the video generation module 802 is specifically used to:

[0162] Based on the video data streams respectively corresponding to at least one interface image, an operation track video of the target user is generated.

[0163] In an optional embodiment, after generating the target user's operation trajectory video based on at least one interface response information and at least one interface touch information respectively associated with the operation, the user operation display device 800 further includes an operation display module 803, and the operation display module 803 is specifically used to:

[0164] In response to an operation track query request for a target user, an operation track video of the target user is screened out from a preset operation track video set; wherein the operation track video set includes: an operation track video corresponding to at least one user;

[0165] Play the target user's operation trajectory video.

[0166] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0167] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0168] An embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute a method according to an embodiment of the present application.

[0169] See also Fig. 9 As shown, the structured block diagram of the electronic device 900 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent the computer device of various forms of digital electronics, such as, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described herein and / or required.

[0170] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0171] A plurality of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 may be any type of device capable of inputting information to the electronic device 900, and the input unit 906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 907 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 may include but is not limited to a disk, an optical disk. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a worldwide interoperability for microwave access (WiMax) device, a cellular communication device, and / or the like.

[0172] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned user operation display method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the above-mentioned user operation display method by any other appropriate means (e.g., by means of firmware).

[0173] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0174] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0176] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tub (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0177] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0178] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0179] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present application.< / view>

Claims

1. A method for displaying user operations, characterized in that: include: In response to at least one operation of a target user on a target operating platform, record interface response information of the target operating platform corresponding to the at least one operation; wherein each interface response information is used to indicate a service provided by the target operating platform for the target user according to the corresponding operation; Based on at least one interface response information and the interface touch information respectively associated with the at least one operation, an operation track video of the target user is generated; wherein the operation track video is used to display the at least one operation.

2. The method according to claim 1, characterized in that The recording of the interface response information corresponding to the target operating platform based on the at least one operation includes: Determining at least one interface information recording method based on the operation type corresponding to the at least one operation; Based on the at least one interface information recording method, interface response information of the target operating platform corresponding to the at least one operation is recorded.

3. The method according to claim 1 or 2, characterized in that The step of generating the operation track video of the target user based on the at least one interface response information and the interface touch information respectively associated with the at least one operation includes: Obtaining root views corresponding to the at least one operation based on the at least one interface response information; wherein each root view is used to indicate the interface framework layout of the target operation platform under the corresponding operation; Based on the interface touch information and the subview set respectively corresponding to at least one root view, draw the interface image respectively corresponding to the at least one operation; Based on the at least one interface image obtained, an operation track video of the target user is generated.

4. The method according to claim 3, characterized in that Drawing the interface image corresponding to the at least one operation based on the interface touch information and the subview set corresponding to the at least one root view respectively includes: For the at least one operation, perform the following operations respectively: Based on a root view corresponding to an operation, determining an interface object corresponding to the operation; wherein the interface object is used to indicate an operation window corresponding to the operation in the target operation platform; Using the touch information acquisition method set for the interface object to obtain corresponding interface touch information; Based on the interface touch information and the subview set of the root view, an interface image of the operation is drawn.

5. The method according to claim 4, characterized in that Drawing the interface image of the operation based on the interface touch information and the subview set of the root view includes: Traversing the sub-view set based on at least one sub-view response included in the interface response information to obtain at least one sub-view corresponding to the one operation; Based on the interface touch information and the at least one sub-view, an interface image of the operation is drawn.

6. The method according to claim 4 or 5, characterized in that After drawing the interface image of the operation based on the interface touch information and the subview set of the root view, the method further includes: Performing video data encoding on the interface image of the one operation to obtain a corresponding video data stream; The step of generating an operation track video of the target user based on the at least one interface image obtained includes: Based on the video data streams respectively corresponding to the at least one interface image, an operation trajectory video of the target user is generated.

7. The method according to claim 1 or 2, characterized in that: After generating the operation track video of the target user based on the at least one interface response information and the interface touch information respectively associated with the at least one operation, the method further includes: In response to an operation track query request for the target user, an operation track video of the target user is screened out from a preset operation track video set; wherein the operation track video set includes: an operation track video corresponding to at least one user; Play the operation track video of the target user.

8. A user-operated display device, characterized in that: include: An information recording module, configured to record, in response to at least one operation of a target user on a target operating platform, interface response information corresponding to the target operating platform based on the at least one operation; wherein each interface response information is used to indicate a service provided by the target operating platform for the target user according to the corresponding operation; The video generation module is used to generate an operation track video of the target user based on at least one interface response information and the interface touch information respectively associated with the at least one operation; wherein the operation track video is used to display the at least one operation.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images