Page operation process backtracking method and device, computer device and storage medium

Abstract

The application relates to a page operation process backtracking method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: in response to a user page operation, a screen recording object is obtained; the screen recording object is recorded; sampling buffer data in the recording process of the screen recording object is obtained; a pixel buffer area reference is created according to the sampling buffer data, an image information object is generated according to the successfully created pixel buffer area reference, the image information object is cached to a preset pixel buffer area, and the image information object comprises information required for generating an image; when it is confirmed that the user page operation is successful, the image information object is obtained from the preset pixel buffer area, the image information object is encoded, and a backtracking video is obtained. The method can realize accurate positioning of a page operation problem.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, and storage medium for tracing back the page operation process. Background Technology

[0002] With the development of computer technology, page operation process backtracking technology has emerged, which is used to trace back the page operation process. For example, the page operation process can specifically refer to the process by which the user performs operations on the terminal page.

[0003] In traditional technologies, the main backtracking method involves collecting static page text data corresponding to each current page during page operations, adding timestamps to each static page text data based on the time of the collected static page text data, storing the timestamped static page text data, and restoring the corresponding pages according to the timestamp order to achieve visual log backtracking of the page operation process.

[0004] However, in traditional methods, static page text data corresponds to the logs of each operation (click, check mark, long press, etc.). When the page operation is to fill in information on the page, only the corresponding text and image data can be obtained. If the page operation is incorrect and causes a program error, it is difficult to locate the page operation problem. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, storage medium, and program product that can accurately locate page operation problems and trace back the page operation process to address the above-mentioned technical issues.

[0006] A method for tracing back the page operation process, the method comprising:

[0007] Respond to user page actions and obtain the screen recording object;

[0008] Record the screen of the object being recorded;

[0009] Acquire sampling buffer data during the recording process of the screen recording object;

[0010] Based on the sampled buffer data, a pixel buffer reference is created. Based on the successfully created pixel buffer reference, an image information object is generated. The image information object is cached in the preset pixel buffer. The image information object includes the information needed to generate the image.

[0011] When the user's page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and the retrospective video is obtained.

[0012] A page operation process retracing device, the device comprising:

[0013] The response module is used to respond to user page operations and obtain screen recording objects;

[0014] The recording module is used to record screen objects.

[0015] The acquisition module is used to acquire sampling buffer data during the recording process of the screen recording object;

[0016] The processing module is used to create a pixel buffer reference based on the sampled buffer data, generate an image information object based on the successfully created pixel buffer reference, and cache the image information object in a preset pixel buffer. The image information object includes the information needed to generate the image.

[0017] The encoding module is used to retrieve image information objects from a preset pixel buffer and encode the image information objects to obtain the retrospective video when the user's page operation is confirmed to be successful.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0019] Respond to user page actions and obtain the screen recording object;

[0020] Record the screen of the object being recorded;

[0021] Acquire sampling buffer data during the recording process of the screen recording object;

[0022] Based on the sampled buffer data, a pixel buffer reference is created. Based on the successfully created pixel buffer reference, an image information object is generated. The image information object is cached in the preset pixel buffer. The image information object includes the information needed to generate the image.

[0023] When the user's page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and the retrospective video is obtained.

[0024] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0025] Respond to user page actions and obtain the screen recording object;

[0026] Record the screen of the object being recorded;

[0027] Acquire sampling buffer data during the recording process of the screen recording object;

[0028] Based on the sampled buffer data, a pixel buffer reference is created. Based on the successfully created pixel buffer reference, an image information object is generated. The image information object is cached in the preset pixel buffer. The image information object includes the information needed to generate the image.

[0029] When the user's page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and the retrospective video is obtained.

[0030] A computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0031] Respond to user page actions and obtain the screen recording object;

[0032] Record the screen of the object being recorded;

[0033] Acquire sampling buffer data during the recording process of the screen recording object;

[0034] Based on the sampled buffer data, a pixel buffer reference is created. Based on the successfully created pixel buffer reference, an image information object is generated. The image information object is cached in the preset pixel buffer. The image information object includes the information needed to generate the image.

[0035] When the user's page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and the retrospective video is obtained.

[0036] The aforementioned page operation process retracing method, apparatus, computer equipment, storage medium, and program product, by responding to user page operations, acquiring a screen recording object, recording the screen recording object, and acquiring sampling buffer data during the recording process, can achieve real-time response to user page operations. It acquires sampling buffer data corresponding to the user page operation, creates pixel buffer references based on the sampling buffer data, generates image information objects based on the successfully created pixel buffer references, and caches the image information objects in a preset pixel buffer. It can manage the sampling buffer data by creating pixel buffer references, and then use the successfully created pixel buffer references to acquire the image information objects corresponding to the user page operation. Therefore, when the user page operation is confirmed to be successful, the image information object is retrieved from the preset pixel buffer, and by encoding the image information object, a retracing video is obtained. The retracing video allows for the retracing of user page operations, restoring the user's operation process when filling in information on the page, and enabling accurate location of page operation problems. Attached Figure Description

[0037] Figure 1This is a diagram illustrating the application environment of a page operation process backtracking method in one embodiment;

[0038] Figure 2 This is a flowchart illustrating a page operation process backtracking method in one embodiment;

[0039] Figure 3 This is a schematic diagram of the terminal environment configuration for a page operation process backtracking method in one embodiment;

[0040] Figure 4 This is an application scenario diagram of the page operation process backtracking method in one embodiment;

[0041] Figure 5 This is an application scenario diagram of the page operation process backtracking method in another embodiment;

[0042] Figure 6 This is a flowchart illustrating the page operation process backtracking method in another embodiment;

[0043] Figure 7 This is a structural block diagram of a page operation process tracing device in one embodiment;

[0044] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The page operation process backtracking method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 responds to user page operations, acquires a screen recording object, records the screen recording object, acquires sampling buffer data during the recording process, creates a pixel buffer reference based on the sampling buffer data, generates an image information object based on the successfully created pixel buffer reference, and caches the image information object in a preset pixel buffer. The image information object includes information needed to generate the image. When the user page operation is confirmed to be successful, the image information object is retrieved from the preset pixel buffer, encoded, and used to obtain a playback video. The playback video is then uploaded to server 104. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a node in a blockchain.

[0047] In one embodiment, such as Figure 2 As shown, a method for backtracking page operation processes is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0048] Step 202: Respond to user page operations and obtain the screen recording object.

[0049] User page operations refer to user actions on the pages displayed on the terminal. Specifically, user page operations can refer to actions such as swiping and clicking. A screen recording object is a singleton object used to record the screen. For example, a screen recording object can be a wrapped RPScreenRecorder (a shared recorder object that provides the ability to record application audio and video).

[0050] Specifically, when a user's interaction with the displayed page is detected, the terminal responds to the user's page interaction by obtaining the screen recording object. For example, when the screen recording object is a wrapped RPScreenRecorder, it can be obtained through the sharedRecorder function, which returns a shared screen recorder instance of the application.

[0051] Step 204: Record the screen recording target.

[0052] Specifically, after obtaining the screen recording object, the terminal calls the "Start Recording" method to begin recording the screen. The "Start Recording" method is used to begin screen recording. For example, the "Start Recording" method could be the `startRecording` function, which is used to begin recording the application display.

[0053] Step 206: Obtain the sampling buffer data during the recording process of the screen recording object.

[0054] Here, sampling buffer data refers to the raw data of the recorded footage. For example, sampling buffer data can specifically refer to sampleBuffer data, which is a CMSampleBuffer object. In this embodiment, sampleBuffer data contains video data.

[0055] Specifically, when recording a screen recording object, the terminal calls the "Get Recorded Image" method, which retrieves the sampling buffer data in a callback. This callback can specifically refer to a block callback, and the callback function can be pre-added to the screen recording object. The "Get Recorded Image" method is used to obtain the sampling buffer data. For example, the "Get Recorded Image" method could specifically be the `startCaptureWithHandler` method, which is used to initiate screen capture.

[0056] Furthermore, the terminal will initiate an asynchronous task, calling the method to acquire recorded images, and retrieving sampling buffer data in the block callback. It's important to note that the block callback can continuously execute within a pre-defined time interval. When the pre-defined time interval is 60 FPS (Frames Per Second), the terminal will call the method 60 times per second, sequentially executing the steps of calling the method to acquire recorded images, retrieving sampling buffer data in the callback, and generating image information objects based on the sampling buffer data. That is, 60 image information objects will be generated per second.

[0057] Step 208: Based on the sampling buffer data, create a pixel buffer reference, generate an image information object based on the successfully created pixel buffer reference, and cache the image information object in a preset pixel buffer. The image information object includes the information needed to generate the image.

[0058] In this context, an image information object refers to an object that contains all the information needed to generate an image. For example, an image information object can specifically refer to a CIImage object. A CIImage object is a representation of an image processed or generated by the core image filter; although it contains image data associated with the image, it is not the image itself. A CIImage object can be considered an image "recipe," containing all the information needed to generate an image. A preset pixel buffer refers to a pre-set pixel buffer used to cache image information objects.

[0059] Specifically, after obtaining the sampling buffer data, the terminal calls the method to capture the sampling buffer to the image buffer. Based on the sampling buffer data, a pixel buffer reference is created, and then an image information object is generated based on the successfully created pixel buffer reference. After obtaining the image information object, the terminal caches the image information object in a preset pixel buffer for later use. Here, a pixel buffer reference is a pixel image type; it is a collection of pixels in an image frame, containing information about each pixel. For example, the pixel information could refer to the starting coordinates of each pixel. For instance, a pixel buffer reference could specifically refer to CVPixelBufferRef; because it starts with CV, it belongs to the CoreVideo module and includes all compressed image information and video frame formats.

[0060] Step 210: When the user page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, the image information object is encoded, and the retrospective video is obtained.

[0061] In this context, "successful user page operation" means that the user has successfully submitted information. For example, when purchasing insurance, "successful user page operation" could specifically mean that the user has successfully ordered insurance.

[0062] Specifically, upon confirming a successful user page operation, the terminal retrieves the image information object from a preset pixel buffer, encodes it, and obtains the playback video. Further, after obtaining the playback video, the terminal uploads it to the server, allowing the server to rewind the page operation process based on the playback video. It should be noted that since the image information objects are stored in the preset pixel buffer in chronological order, encoding them directly according to their entry order is sufficient. If the user page operation is unsuccessful, the terminal directly removes the bitmap data from the pixel buffer.

[0063] For example, in this embodiment, H.264 encoding can be used to generate the video stream. H.264 is a highly compressed digital video codec standard proposed by the Joint Video Team (JVT), which is composed of the Video Coding Experts Group (VCEG) of ITU-T (ITU-T for ITU Telecommunication Standardization Sector) and the Moving Picture Experts Group (MPEG) of ISO / IEC (International Organization for Standardization).

[0064] The aforementioned page operation process retracing method, by responding to user page operations, obtains a screen recording object, records the screen recording object, and acquires sampling buffer data during the recording process, can achieve real-time response to user page operations. It obtains sampling buffer data corresponding to the user page operation, creates a pixel buffer reference based on the sampled buffer data, generates an image information object based on the successfully created pixel buffer reference, and caches the image information object in a preset pixel buffer. This allows for the management of sampling buffer data through pixel buffer reference creation, and further enables the acquisition of the image information object corresponding to the user page operation using the successfully created pixel buffer reference. Therefore, when the user page operation is confirmed to be successful, the image information object is retrieved from the preset pixel buffer, encoded, and used to obtain a retracing video. This retracing video allows for the retracing of user page operations, reconstructing the user's operation process when filling in information on the page, and enabling accurate localization of page operation problems.

[0065] In one embodiment, creating a pixel buffer reference based on the sampled buffer data, and generating an image information object based on the successfully created pixel buffer reference, includes:

[0066] Call the method to capture the sampling buffer to the image buffer, and create a pixel buffer reference based on the sampling buffer data;

[0067] When the pixel buffer reference is successfully created, an image information object is generated based on the pixel buffer reference and the preset attribute information of the preset pixel buffer.

[0068] The "Capture Sample Buffer to Image Buffer" method is used to create a pixel buffer reference based on the sample buffer data. Specifically, this method could be `CMSampleBufferGetImageBuffer`, which returns `CVImageBuffer` media data (i.e., a pixel buffer reference) based on `CMSampleBuffer` (the sample buffer data). It's important to note that if `CMSampleBuffer` does not contain `CVImageBuffer`, contains `CMBlockBuffer`, or there are other errors resulting in a null return value (NULL), a pixel buffer reference cannot be created. Therefore, when calling the "Capture Sample Buffer to Image Buffer" method to create a pixel buffer reference based on the sample buffer data, it's necessary to check if the pixel buffer reference is null. Only if the pixel buffer reference is not null can the next step of processing proceed.

[0069] The preset attribute information refers to the pre-configured attributes of the preset pixel buffer, including the type of the pixel buffer, whether it has the ability to generate bitmaps, the width of the pixels, and the height of the pixels. For example, the type of the pixel buffer is represented by kCVPixelBufferPixelFormatTypeKey, and the corresponding preset attribute information could be kCVPixelFormatType_32BGRA, indicating a 32-bit RGBA pixel. Whether it has the ability to generate bitmaps is represented by kCVPixelBufferCGBitmapContextCompatibilityKey, and the specific preset attribute information could be that it generates bitmaps using 32-bit pixels. The width of the pixels is represented by kCVPixelBufferWidthKey, and the height of the pixels is represented by kCVPixelBufferHeightKey. Based on the preset attribute information, the terminal can call the CVPixelBufferPoolCreate method to create a CVPixelBufferPoolRef. CVPixelBufferPoolRef is a technology in the CoreVideo framework used to manage CVPixelBufferRefs. It can manipulate the pixel buffers in the image and various memory storage methods for the image. Using this interface, it is possible to manipulate YUV (a color encoding technology), pixels, and the memory space they occupy.

[0070] Specifically, the terminal calls the method to capture the sampling buffer to the image buffer. Based on the sampling buffer data, it attempts to create a pixel buffer reference. When the sampling buffer data contains a CVImageBuffer (a reference to the core video image buffer), the pixel buffer reference can be created successfully (i.e., it is not empty). The terminal then further generates an image information object based on the pixel buffer reference and the preset attribute information of the pixel buffer. The pixel buffer reference contains information about all pixels, including the coordinates of each pixel. The terminal can then fill in the pixels of the image based on the pixel buffer reference and the preset attribute information, thus initializing the image information object.

[0071] In this embodiment, by calling the capture sampling buffer to image buffer method, a pixel buffer reference can be created based on the sampling buffer data. Thus, when the pixel buffer reference is successfully created, an image information object can be generated based on the pixel buffer reference and the preset attribute information of the preset pixel buffer.

[0072] In one embodiment, after creating a pixel buffer reference based on the sampled buffer data and generating an image information object based on the successfully created pixel buffer reference, the method further includes:

[0073] Get the bitmap context corresponding to the image information object;

[0074] The image information object is rendered according to the bitmap context to obtain bitmap data;

[0075] Write bitmap data to a preset image buffer.

[0076] In this context, the bitmap context refers to the basic information used to convert image information objects into UI (User Interface) images. Bitmap data refers to the user interface image produced based on the bitmap context and image information objects. A semaphore is a facility used in a multithreaded environment to ensure that two or more critical code segments are not called concurrently. Before entering a critical code segment, a thread must acquire a semaphore; once the critical code segment is completed, the thread must release the semaphore. A pre-defined image buffer refers to a pre-set area for storing bitmap data.

[0077] Specifically, after generating the image information object based on the sampled buffer data, the terminal calls a context creation function to obtain the bitmap context corresponding to the image information object. It then renders the image information object based on the bitmap context to complete the drawing of the bitmap data, obtaining the bitmap data. Next, it initiates bitmap writing in the semaphore control queue to ensure that bitmap writing is not concurrently called, and finally writes the bitmap data to a preset image buffer. The context creation function can specifically refer to the `UIGraphicsBeginImageContextWithOptions` function, which can create a bitmap-based graphics context using specified options. Furthermore, before obtaining the bitmap context, the terminal also encapsulates a pixel cache creation function to manage the obtained bitmap context. For example, the terminal can use functions related to `RPScreenRecorder` for this encapsulation.

[0078] For example, rendering an image information object based on a bitmap context can be done as follows: The terminal first calls the `UIGraphicsPushContext` function, passing in the bitmap context, then calls the `UIImage.drawInRect` method to draw the image frame based on the bitmap context and the image information object, obtaining the bitmap data. Finally, the `CGContextRelease` method is called to release the bitmap context memory. Specifically, the `UIGraphicsPushContext` function makes the specified graphics context (in this example, the bitmap context) the current context, and the `drawInRect` method draws the entire image within the specified rectangle (the coordinate system of the bitmap context), scaling it as needed.

[0079] In this embodiment, by obtaining the bitmap context corresponding to the image information object, the image information object can be rendered according to the bitmap context to obtain bitmap data. By enabling bitmap writing in the semaphore control queue, bitmap data can be written to the preset image buffer while ensuring that bitmap writing is not concurrently called, thus achieving correct writing of bitmap data.

[0080] In one embodiment, the page operation process backtracking method further includes:

[0081] When it is confirmed that the user's page operation has failed, generate and display a prompt to retrieve bitmap data;

[0082] When a consent response corresponding to the prompt to retrieve bitmap data is received, the bitmap data is retrieved from the preset image buffer;

[0083] Upload a problem location message containing bitmap data to the server. The problem location message is used to help the server confirm the reason for the operation failure based on the bitmap data.

[0084] Specifically, when a user's page operation fails, both the terminal and the server need to confirm the reason for the failure. At this point, the terminal generates and displays a prompt to retrieve bitmap data. This prompt informs the user that the terminal wants to retrieve bitmap data and upload it to the server. When the terminal receives a confirmation response corresponding to the bitmap data retrieval prompt, it indicates that the user authorizes the terminal to retrieve and upload the bitmap data to the server. The terminal then retrieves the bitmap data from a pre-defined image buffer and uploads a problem location prompt carrying the bitmap data to the server, prompting the server to confirm the reason for the failure based on the bitmap data. Furthermore, after confirming the reason for the failure, the server sends feedback to the terminal, allowing the user to confirm the cause of the failure and avoid repeating the same mistake on future page operations.

[0085] In this embodiment, upon receiving consent feedback, bitmap data is acquired and a problem location prompt carrying the bitmap data is uploaded to the server. This prompts the server to confirm the reason for the operation failure based on the bitmap data, enabling the server to send the corresponding reason for the operation failure to the terminal. This allows the user of the terminal to confirm the reason for the operation failure so as to avoid making the same mistake again when performing page operations in the next time. This achieves problem location and improves the success rate of user page operations.

[0086] In one embodiment, after creating a pixel buffer reference based on the sampled buffer data, generating an image information object based on the successfully created pixel buffer reference, and caching the image information object to a preset pixel buffer, the method further includes:

[0087] If no call to the stop recording method is detected, return to the step of recording the screen recording object;

[0088] When the time interval between user page operations is detected to be greater than the preset operation time threshold, and / or a click event of the target tracking point is detected, the user operation is confirmed to have ended, and the stop recording method is called.

[0089] The `stop recording` method is used to stop recording of the screen recording object. For example, the `stopCaptureWithHandler` method specifically stops screen capture. The user page operation time interval refers to the time between one page operation and the next; the preset operation time threshold can be set as needed. Target tracking points refer to pre-set tracking points corresponding to page confirmation / submission information. For example, target tracking points can refer to tracking points corresponding to events such as submit and confirm.

[0090] Specifically, when the terminal confirms the end of the user's operation, it needs to call the "Stop Recording" method. If no call to the "Stop Recording" method is detected, it indicates that the user's operation has not ended, and the terminal will return to the step of calling the "Start Recording" method to record the screen recording object, and re-acquire the image information object until the "Stop Recording" method is detected, indicating that the user's operation has ended. The terminal will then generate the corresponding retrospective video based on all image information objects in the preset pixel buffer. The terminal can confirm the end of the user's operation through preset operations, which can be set as needed. For example, the preset operation could be that the user has not interacted with the screen for a certain period of time, or that the user has confirmed and submitted information on the page. The specified time period can be set as needed.

[0091] The terminal can monitor recording attributes by adding monitoring methods, detecting changes in these attributes to trigger a stop recording method. For example, adding a monitoring method could refer to the `addObserver` method, and monitoring recording attributes could specifically involve monitoring the `recording` property of `RPScreenRecorder`. Furthermore, monitoring recording attributes primarily focuses on changes in these attributes. Therefore, in addition to detecting the end of recording, it can also detect recording interruptions, such as an incoming phone call during user interaction. Upon detecting an interruption, the terminal will pause recording, wait for the interruption to resolve, and then resume recording.

[0092] Specifically, the terminal will monitor in real time whether the user's page operation has ended. The end of the user's page operation can be defined as the user not interacting with the screen for a certain period of time or confirming and submitting information on the page. Therefore, when the terminal detects that the time interval of the user's page operation is greater than the preset operation time threshold (indicating that the user has not interacted with the screen for a certain period of time) and / or the click event of the target tracking point (indicating that the user has confirmed and submitted information on the page), it will confirm that the user operation has ended and call the stop recording method to stop recording.

[0093] In this embodiment, by monitoring the time interval of user page operations and the click events of target tracking points, it is possible to confirm the end of user operations. Thus, upon confirming the end of user operations, the stop recording method can be called to stop recording.

[0094] In one embodiment, after confirming successful user page operation, the process includes retrieving an image information object from a preset pixel buffer, encoding the image information object to obtain the playback video, and then further including:

[0095] Obtain the operation information corresponding to the user's page operation, and upload the operation information and the playback video to the server.

[0096] Operation information refers to the information entered by the user when performing actions on the page. For example, when purchasing insurance, operation information could specifically refer to the form data entered by the user.

[0097] Specifically, when a user interacts with a page, the terminal records the corresponding event tracking points. After obtaining the playback video, the terminal uses the recorded event tracking points to retrieve the corresponding operation information and uploads both the operation information and the playback video to the server. Furthermore, when uploading the operation information and playback video to the server, the terminal generates a local URL (Uniform Resource Locator) address, uploads the operation information to the server, and asynchronously uploads the playback video to the server. It should be noted that if the playback video upload fails, it will be retried the next time the user interacts with the page.

[0098] In this embodiment, by obtaining the operation information corresponding to the user's page operation and uploading the operation information and the retrospective video to the server, the server can retrospectively analyze the user's page operation based on the retrospective video and confirm the user information based on the operation information.

[0099] In one embodiment, the page operation process backtracking method further includes:

[0100] Send a request to retrieve page data;

[0101] Receive page data corresponding to the page data retrieval request;

[0102] Generate and display the page to be operated on based on the page data.

[0103] Among them, the page data retrieval request is a request to the server to retrieve page data, which refers to the data used to generate the page to be displayed.

[0104] Specifically, when a page operation is required, the user sends a page data retrieval request to the server through the terminal. After receiving the page data retrieval request, the server sends the page data corresponding to the page data retrieval request to the terminal. The terminal can then generate and display the page to be operated based on the received page data corresponding to the page data retrieval request.

[0105] For example, when the page operation involves purchasing insurance, the user will enter the insurance purchase page on the terminal and fill in their user information. The terminal will then send a page data retrieval request to the server based on the user information. The server will then send the corresponding insurance form page data based on the user information. After receiving the insurance form page data, the terminal will create a form serialization parser, parsing different insurance items for different users. Information such as the coverage plan, insurance details, application instructions, and information disclosure will be converted into fields in JSON (JavaScript Object Notation) data to establish data relationships. The `toDataObj` method provided by the application will be used to convert the JSON field values ​​into object types, creating a `tableView`. Form cells will be generated based on the object data, and further processed based on these form cells to generate the page to be operated. Furthermore, after the form is created, the terminal will configure the current data environment. For example, the configuration on the iOS side can be as follows: Figure 3 As shown.

[0106] In this embodiment, by sending a page data acquisition request, receiving page data corresponding to the page data acquisition request, generating and displaying the page to be operated based on the page data, the display of the page to be operated can be realized.

[0107] This application also provides an application scenario for order confirmation, which applies the aforementioned page operation process backtracking method. For example... Figure 4 As shown, specifically, the application of this page operation process backtracking method in this application scenario is as follows:

[0108] The terminal sends a form page data retrieval request to the server. The server then sends the form page data to the terminal. The terminal loads the form page data, creates a parser to generate a form view (i.e., the page to be operated), receives user form operation behavior, initializes the recording system (i.e., responds to user page operations, obtains the screen recording object, and records the screen recording object), starts a queue and creates a pixel buffer. In the block callback, each frame of data is written to the graphics buffer (obtains the sampling buffer data during the recording process of the screen recording object, creates a pixel buffer reference based on the sampling buffer data, generates an image information object based on the successfully created pixel buffer reference, and caches the image information object in the preset pixel buffer), generates an order, ends recording, encodes it into a video stream, and establishes a correspondence with the order (when the user page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, encoded, and the playback video is obtained), and uploads the video stream and the correspondence with the order to the server (i.e., uploads the operation information and the playback video to the server).

[0109] This application also provides another application scenario: implementing insurance purchase on an iOS-based terminal. This application scenario utilizes the aforementioned page operation process backtracking method. For example... Figure 5 As shown, specifically, the application of this page operation process backtracking method in this application scenario is as follows:

[0110] When a user accesses the insurance purchase page via their terminal, the terminal sends a page data retrieval request carrying the user's information to the server. The server then issues insurance form data based on the user's information. The terminal receives the insurance form data, parses it into a data object using a parser, generates a form view, configures the data environment, adds an observer pattern to monitor the recording property of RPScreenRecorder, calls the sharedRecorder of the application's (i.e., TMF in the diagram) basic recording object to obtain the screen recording object, calls startRecording (start recording method) to begin recording the screen recording object, and presets the kCVPixelBuffer (pixel buffer) property (i.e., preset attribute information of the pixel buffer).

[0111] The serial queue is started, and asynchronous rendering is initiated via `dispatch_async`. The terminal calls `startCaptureWithHandler` (the method to get the recorded image), and in the block callback, it obtains the `sampleBuffer` (sample buffer data). It then calls `CMSampleBufferGetImageBuffer` (the method to capture the sample buffer to the image buffer) and passes the `sampleBuffer` to attempt to create a `CVPixelBufferRef` (pixel buffer reference). When `CVPixelBufferRef` is not null (i.e., the pixel buffer reference is successfully created), it initializes a `CIImage` object (image information object) based on `CVPixelBufferRef`, caches the `CIImage` object in a preset pixel buffer, and the `CIImage` object includes the information needed to generate the image. A bitmap context is generated, and `UIGraphicsPushContext` is called, passing the bitmap context. Finally, `UIImage's drawInR()` is called. The `ect` method draws the image frame, calls `CGContextRelease` to release the bitmap context memory (i.e., renders the image information object according to the bitmap context to obtain bitmap data), starts bitmap writing in the semaphore control queue, writes bitmap data to the preset image buffer, checks whether `stop` is called, and if the call is not finished, returns to call `startRecording` (start recording method) to start recording the screen recording object. When the call is finished, the callback of the `recording` property is executed (i.e., if no call to stop recording method is detected, the step of recording the screen recording object is returned; if the user page operation time interval is detected to be greater than the preset operation time threshold and / or the click event of the target point is detected, the user operation is confirmed to be finished and the stop recording method is called). When the user page operation is successful, video encoding is performed and data is uploaded (i.e., the image information object is obtained from the preset pixel buffer, the image information object is encoded to obtain the backtracking video, the operation information corresponding to the user page operation is obtained, and the operation information and backtracking video are uploaded to the server).

[0112] In one embodiment, such as Figure 6 As shown, this application also provides a flowchart to illustrate the above-mentioned page operation process backtracking method, which specifically includes the following steps:

[0113] Step 602: Send a page data retrieval request;

[0114] Step 604: Receive the page data corresponding to the page data retrieval request;

[0115] Step 606: Generate and display the page to be operated based on the page data;

[0116] Step 608: Respond to user page operation and obtain the screen recording object;

[0117] Step 610: Record the screen recording target;

[0118] Step 612: Obtain sampling buffer data during the recording process of the screen recording object;

[0119] Step 614: Call the method to capture the sampling buffer to the image buffer, and create a pixel buffer reference based on the sampling buffer data;

[0120] Step 616: When the pixel buffer reference is successfully created, an image information object is generated based on the pixel buffer reference and the preset attribute information of the preset pixel buffer. The image information object is cached in the preset pixel buffer. The image information object includes the information required to generate the image.

[0121] Step 618: Obtain the bitmap context corresponding to the image information object;

[0122] Step 620: Render the image information object according to the bitmap context to obtain bitmap data;

[0123] Step 622: Write the bitmap data into the preset image buffer;

[0124] Step 624: If no call to the stop recording method is detected, return to the step of recording the screen recording object;

[0125] Step 626: When the time interval between user page operations is detected to be greater than the preset operation time threshold and / or the click event of the target tracking point, confirm that the user operation has ended and call the stop recording method;

[0126] Step 628: Determine whether the user's page operation was successful. If the user's page operation is confirmed to have failed, proceed to step 630. If the user's page operation is confirmed to have succeeded, proceed to step 636.

[0127] Step 630: Generate and display the prompt for obtaining bitmap data;

[0128] Step 632: When a consent feedback corresponding to the prompt to obtain bitmap data is received, bitmap data is obtained from the preset image buffer;

[0129] Step 634: Upload the problem location prompt carrying bitmap data to the server. The problem location prompt is used to help the server confirm the reason for the operation failure based on the bitmap data.

[0130] Step 636: Obtain the image information object from the preset pixel buffer, encode the image information object, and obtain the retrospective video;

[0131] Step 638: Obtain the operation information corresponding to the user's page operation, and upload the operation information and the playback video to the server.

[0132] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0133] In one embodiment, such as Figure 7 As shown, a page operation process tracing device is provided. This device can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the device includes: a response module 702, a recording module 704, an acquisition module 706, a processing module 708, and an encoding module 710, wherein:

[0134] Response module 702 is used to respond to user page operations and obtain screen recording objects;

[0135] Recording module 704 is used to record the screen recording object;

[0136] The acquisition module 706 is used to acquire sampling buffer data during the recording process of the screen recording object;

[0137] Processing module 708 is used to create a pixel buffer reference based on the sampling buffer data, generate an image information object based on the successfully created pixel buffer reference, and cache the image information object in a preset pixel buffer. The image information object includes the information required to generate the image.

[0138] The encoding module 710 is used to obtain an image information object from a preset pixel buffer and encode the image information object to obtain the retrospective video when the user page operation is confirmed to be successful.

[0139] The aforementioned page operation process retracing device, by responding to user page operations, acquires the screen recording object, records the screen recording object, and obtains sampling buffer data during the recording process, can achieve real-time response to user page operations. It acquires sampling buffer data corresponding to the user page operation, creates pixel buffer references based on the sampled buffer data, generates image information objects based on the successfully created pixel buffer references, and caches the image information objects in a preset pixel buffer. This allows for the management of sampling buffer data through the creation of pixel buffer references, and further enables the acquisition of image information objects corresponding to user page operations. Therefore, when the user page operation is confirmed to be successful, the image information object is retrieved from the preset pixel buffer, encoded, and used to obtain a retracing video. This retracing video allows for the retracing of user page operations, reconstructing the user's operation process when filling in information on the page, and enabling accurate location of page operation problems.

[0140] In one embodiment, the processing module is further configured to call the capture sampling buffer to image buffer method, create a pixel buffer reference based on the sampling buffer data, and generate an image information object based on the pixel buffer reference and the preset attribute information of the preset pixel buffer when the pixel buffer reference is successfully created.

[0141] In one embodiment, the page operation process tracing device further includes a drawing module, which is used to obtain the bitmap context corresponding to the image information object, render the image information object according to the bitmap context to obtain bitmap data, and write the bitmap data into a preset image buffer.

[0142] In one embodiment, the page operation process backtracking device further includes an upload module. The upload module is used to generate and display a prompt to obtain bitmap data when it is confirmed that the user's page operation has failed. When it receives the consent feedback corresponding to the prompt to obtain bitmap data, it obtains bitmap data from a preset image buffer and uploads a problem location prompt carrying the bitmap data to the server. The problem location prompt is used to prompt the server to confirm the reason for the operation failure based on the bitmap data.

[0143] In one embodiment, the page operation process backtracking device further includes a monitoring module. The monitoring module is used to return to the steps of recording the screen recording object when no call to the stop recording method is detected, and to confirm the end of the user operation and call the stop recording method when the user page operation time interval is greater than a preset operation time threshold and / or a click event of the target point is detected.

[0144] In one embodiment, the upload module is also used to obtain operation information corresponding to the user's page operation, and upload the operation information and the playback video to the server.

[0145] In one embodiment, the page operation process tracing device further includes a page generation module, which is used to send a page data acquisition request, receive page data corresponding to the page data acquisition request, generate and display the page to be operated based on the page data.

[0146] Specific limitations regarding the page operation process backtracking device can be found in the limitations of the page operation process backtracking method described above, and will not be repeated here. Each module in the aforementioned page operation process backtracking device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0147] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a page operation process rollback method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0148] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0150] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0151] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for tracing back the page operation process, characterized in that, The method includes: Respond to user page actions and obtain the screen recording object; Record the screen recording object; Acquire sampling buffer data during the recording process of the screen recording object; Call the method to capture the sampling buffer to the image buffer, and create a pixel buffer reference based on the sampling buffer data; When the pixel buffer reference is successfully created, an image information object is generated based on the pixel buffer reference and the preset attribute information of the preset pixel buffer. The image information object is then cached in the preset pixel buffer. The image information object includes the information required to generate the image. When the user's page operation is confirmed to be successful, the image information object is obtained from the preset pixel buffer, and the image information object is encoded to obtain the retrospective video.

2. The method according to claim 1, characterized in that, After creating a pixel buffer reference based on the sampled buffer data, and generating an image information object based on the successfully created pixel buffer reference, the process further includes: Obtain the bitmap context corresponding to the image information object; The image information object is rendered according to the bitmap context to obtain bitmap data; The bitmap data is written to a preset image buffer.

3. The method according to claim 2, characterized in that, Also includes: When it is confirmed that the user's page operation has failed, generate and display a prompt to retrieve bitmap data; When a consent feedback corresponding to the prompt to obtain bitmap data is received, bitmap data is obtained from the preset image buffer; Upload a problem location prompt carrying bitmap data to the server. The problem location prompt is used to prompt the server to confirm the reason for the operation failure based on the bitmap data.

4. The method according to claim 1, characterized in that, The process of creating a pixel buffer reference based on the sampled buffer data, generating an image information object based on the successfully created pixel buffer reference, and caching the image information object in a preset pixel buffer further includes: If no call to the stop recording method is detected, return to the step of recording the screen recording object; When the time interval between user page operations is detected to be greater than the preset operation time threshold, and / or a click event of the target tracking point is detected, the user operation is confirmed to have ended, and the stop recording method is called.

5. The method according to claim 1, characterized in that, After confirming the user's page operation is successful, the process of retrieving the image information object from the preset pixel buffer, encoding the image information object to obtain the retrospective video, further includes: Obtain the operation information corresponding to the user's page operation, and upload the operation information and the retrospective video to the server.

6. The method according to claim 1, characterized in that, Also includes: Send a request to retrieve page data; Receive page data corresponding to the page data retrieval request; The page to be operated is generated and displayed based on the page data.

7. A page operation process retracing device, characterized in that, The device includes: The response module is used to respond to user page operations and obtain screen recording objects; A recording module is used to record the screen recording object; The acquisition module is used to acquire sampling buffer data during the recording process of the screen recording object; The processing module is used to call the capture sampling buffer to image buffer method, create a pixel buffer reference according to the sampling buffer data, and when the pixel buffer reference is successfully created, generate an image information object according to the pixel buffer reference and the preset attribute information of the preset pixel buffer, and cache the image information object in the preset pixel buffer. The image information object includes the information required to generate the image. The encoding module is used to obtain the image information object from the preset pixel buffer and encode the image information object to obtain the retrospective video when the user page operation is confirmed to be successful.

8. The apparatus according to claim 7, characterized in that, The page operation process backtracking device also includes a drawing module, which is used to obtain the bitmap context corresponding to the image information object, render the image information object according to the bitmap context to obtain bitmap data, and write the bitmap data into a preset image buffer.

9. The apparatus according to claim 8, characterized in that, The page operation process backtracking device also includes an upload module. The upload module is used to generate and display a prompt to obtain bitmap data when it is confirmed that the user's page operation has failed. When it receives the consent feedback corresponding to the prompt to obtain bitmap data, it obtains bitmap data from the preset image buffer and uploads a problem location prompt carrying bitmap data to the server. The problem location prompt is used to prompt the server to confirm the reason for the operation failure based on the bitmap data.

10. The apparatus according to claim 7, characterized in that, The page operation process backtracking device also includes a monitoring module. The monitoring module is used to return to the step of recording the screen recording object when no call to the stop recording method is detected. When the user page operation time interval is greater than a preset operation time threshold and / or a click event of the target point is detected, the user operation is confirmed to be over and the stop recording method is called.

11. The apparatus according to claim 7, characterized in that, The page operation process retracing device also includes an upload module, which is used to obtain operation information corresponding to the user's page operation and upload the operation information and the retracing video to the server.

12. The apparatus according to claim 7, characterized in that, The page operation process traceability device also includes a page generation module, which is used to send a page data acquisition request, receive page data corresponding to the page data acquisition request, generate and display the page to be operated based on the page data.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images