Backtracking method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a backtracking method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an event stream of interface change, and generating an interface change result record according to the event stream; acquiring a media record when the interface is changed based on a communication module of the browser; sending the interface change result record and the media record to a server; the server is used for fusing the interface change result record and the media record into a media verification record; and in response to a verification instruction, performing backtracking according to the media verification record. By adopting the method, backtracking can be accurately and efficiently carried out, so that verification is facilitated.

Description

Technical Field

[0001] This application relates to the field of backtracking technology, and in particular to a backtracking method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of image technology, the demands for human-computer interaction are also increasing. To ensure the normal use of products and guarantee user experience, it is often necessary to backtrack on the application to determine user behavior.

[0003] However, traditional technologies that use logs for recording and restoration only support static page records, resulting in low accuracy in restoration. Summary of the Invention

[0004] Therefore, it is necessary to provide an accurate and efficient backtracking method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a backtracking method, including:

[0006] Obtain the event stream of interface changes, and generate an interface change result record based on the event stream;

[0007] The browser-based communication module acquires media recordings when the interface changes.

[0008] The interface change result record and the media record are sent to the server; the server is used to merge the interface change result record and the media record into a media verification record;

[0009] In response to the verification instruction, backtracking is performed based on the media verification record.

[0010] In one embodiment, the browser-based communication module acquires media recordings during interface changes, including:

[0011] When the interface of the application to be traced is adjusted in the event stream triggered by the application to be traced, the media data collected by the device is obtained based on the browser's communication module;

[0012] The step of sending the interface change result record and the media record to the server includes:

[0013] The interface change result record and the media data are sent to the server; the server is used to merge the interface change result record and the media data into a media verification record.

[0014] In one embodiment, sending the interface change result record and the media record to the server includes:

[0015] Convert the media data into a media stream;

[0016] The interface change results are recorded and sent to the server along with the media stream;

[0017] The server is used to store the media stream as a media file and record the interface change results and the media file as a media verification record; the metadata associated with the media stream is stored in a database, and the metadata includes the storage path of the media verification record.

[0018] In one embodiment, the server is used to create a virtual desktop and a virtual sound card, and the media verification record includes a playback screen and a playback audio, the media record corresponding to a media file;

[0019] The server is used to place the screen corresponding to the interface change result record and the screen contained in the media file on the virtual desktop, and record the virtual desktop to obtain the retrospective screen; the retrospective screen includes the screen of the media file at each moment, and the screen composed of the behavior content of each behavior position corresponding to each moment of the interface change result record.

[0020] The server is used to record the audio corresponding to the media file through the virtual sound card to obtain the retrospective audio; the retrospective audio includes the audio of the media file at each of the stated times.

[0021] In one embodiment, the media verification record includes behavior type, playback screen, and playback audio; different operation events for different behavior types correspond to different tags;

[0022] The backtracking based on the media verification records includes:

[0023] A timeline for backtracking is displayed, with multiple points in the timeline marked, each mark representing the type of behavior at each point in time;

[0024] Play the flashback footage and the flashback audio according to the instructions for the timeline.

[0025] In one embodiment, acquiring the event stream of interface changes and generating interface change result records based on the event stream includes:

[0026] Obtain the operation sequence triggered by the interface, and generate the event stream of the interface based on the operation sequence and the interface change events triggered by the operation sequence;

[0027] Persist the document object model corresponding to the event stream to obtain a record of the interface change results;

[0028] The interface change result record includes the operation sequence arranged in chronological order and the interface change events triggered by the operation sequence.

[0029] Secondly, this application also provides a backtracking device, comprising:

[0030] The interface module is used to acquire the event stream of interface changes and generate interface change result records based on the event stream.

[0031] The media module is used to acquire media records when the interface changes, based on the browser's communication module.

[0032] The fusion module is used to send the interface change result record and the media record to the server; the server is used to fuse the interface change result record and the media record into a media verification record;

[0033] The backtracking module is used to perform backtracking based on the media verification record in response to the verification command.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the backtracking steps described in any of the above embodiments.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the backtracking steps in any of the above embodiments.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the backtracking steps in any of the above embodiments.

[0037] The aforementioned backtracking method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire an event stream of interface changes, generate interface change result records based on the event stream, and form corresponding documents for events and results caused by user operations. This process is implemented for interfaces with backtracking requirements, reducing resource consumption and helping to reduce user privacy leaks. When using event streams to generate corresponding interface change result records, it is not necessary to record the entire screen directly, and recording can be done at the moment the event stream occurs, allowing for efficient processing with fewer resources. When the interface changes, media records are acquired based on the browser's communication module, which can overcome the platform rule restrictions of the system program, ensure the integrity and stability of media data during interface changes, avoid the cross-platform adaptation work required for application development, and improve the convenience and universality of the verification process. Finally, the interface change result records and the media records are sent to a server. The server is used to merge the interface change result records and the media records into a media verification record, so as to achieve efficient processing of the corresponding data through the server and avoid the recording process before backtracking affecting the user's normal use. Finally, in response to the verification command, a backtracking process is performed based on the media verification records. This process combines static interface change results records with dynamic media verification records for comprehensive analysis, ensuring the accuracy of the backtracking. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a diagram illustrating the application environment of the backtracking method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating the backtracking method in one embodiment;

[0041] Figure 3 This is a flowchart illustrating the media recording acquisition and transmission steps in one embodiment;

[0042] Figure 4 This is a structural block diagram of the backtracking device in one embodiment;

[0043] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] The backtracking method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0046] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In one exemplary embodiment, such as Figure 2 As shown, a backtracking method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0048] Step 202: Obtain the event stream of interface changes and generate an interface change result record based on the event stream.

[0049] The user interface (UI) is the page through which an application interacts with humans and computers. The UI can be used to convey information from the application to the user, and can also be used to receive user operation instructions through UI elements, process the operation instructions, and obtain the corresponding event stream.

[0050] An event flow is an arrangement of events triggered by the interface in chronological order. The event flow includes user actions that trigger events on the interface and affect it; when the terminal detects an event, it calls the corresponding processing function in the application to change the displayed interface elements. Specifically, the event flow can include user events directly triggered by the user, as well as interface change events that trigger changes to the page content or structure; user events include, but are not limited to, mouse movement, clicks, and keyboard input; interface change events include the addition, deletion, or modification of page elements.

[0051] The interface change result record should include at least the event flow and the interface change result triggered by the event flow, so that the event flow, the content and structure of the interface can be traced back through the event recording. The interface change result record may also include the nesting relationship of the changed interface elements, the attributes of the changed interface elements, the content or state of the changed interface elements, and the size and position of the changed interface elements, so as to restore the change process of the corresponding interface in more detail.

[0052] In some embodiments, in response to a recording instruction for backtracking and logging, an event stream of interface changes is acquired, and an interface change result record is generated based on the event stream. Thus, in scenarios requiring logging, dynamic change information on the interface can be recorded, and this process, obtained through the interface change result record, involves relatively little data and has high processing efficiency.

[0053] In some embodiments, obtaining an event stream of interface changes and generating an interface change result record based on the event stream includes: reading user events caused by user operations and interface change events generated by user event adjustments in chronological order to obtain an event stream of interface changes; and recording the event stream based on at least one of the log and document object model corresponding to the event stream to obtain an interface change result record.

[0054] In one example, a user clicks a button, and the resulting UI change event is the display of a pop-up window, thus forming two events in the event flow. Correspondingly, the Document Object Model (DOM) will also change based on these two events, and the DOM snapshot obtained from the DOM change is a record of the UI change result.

[0055] Step 204: Obtain media recordings when the interface changes using the browser's communication module.

[0056] A browser's communication module is a program module integrated into the browser. This module may include interfaces (APIs) under a communication protocol. Because browsers can overcome platform limitations imposed by system programs and support terminal devices on systems such as Windows, macOS, iOS, and Android, obtaining media data through the browser's communication module ensures video integrity and audio transmission stability when the interface changes. This avoids the cross-platform adaptation work required for application development, shortening the development cycle and reducing maintenance costs. Furthermore, users do not need to install applications for media interaction, improving the convenience and universality of the verification process. The browser's communication module can be a collection of programs that support Web Real-Time Communication (WebRTC).

[0057] Media recordings include recordings made by the terminal based on data collected by its external devices, used to reflect information conveyed by the user through multimedia technology when the interface changes. Media recordings include, but are not limited to, video recordings captured by a camera and audio recordings captured by a microphone.

[0058] In some embodiments, the browser-based communication module acquires media records during interface changes, including: in response to a recording instruction for backtracking, during the interface change process, the browser-based communication module collects media data collected by an external device to obtain media records during the interface change. Thus, for scenarios requiring backtracking, not only is data on the interface collected, but also corresponding media data, to complete an efficient acquisition process before backtracking.

[0059] Step 206: Send the interface change result record and media record to the server; the server is used to merge the interface change result record and media record into a media verification record.

[0060] Media verification logs are data fused by the server. They include video or audio recordings based on interface change results and the information contained in each media record. Since event streams reflect user actions, and media records already involve audio and video data processing, further fusion of these two types of data by the terminal might affect user efficiency. Therefore, the server performs the data fusion task to more quickly meet user activity needs.

[0061] Media verification records are the result of the collaboration between page behavior retrospection and real-time audio and video acquisition modules. They can reflect user operations on the interface and audio and video acquisition outside the interface, thus forming a full-scene recording system inside and outside the interface to quickly respond to event needs.

[0062] In some embodiments, the interface change result record and the media record can be sent to the server separately; alternatively, the interface change result record and the media record of the same user can be sent to the server simultaneously.

[0063] Step 208: In response to the verification instruction, backtrack based on the media verification record.

[0064] The verification command is an operation command used to invoke the media verification record. This verification command is obtained by the terminal.

[0065] In some embodiments, if a verification instruction representing a target user is detected, the terminal retrieves a media verification record from the server based on the target user; the terminal then plays the media verification record to achieve backtracking. During the playback of the media verification record, the progress of each node can be determined using snapshots of the document object model.

[0066] In some embodiments, both the interface change result record and the media record are recorded sequentially at each moment, and the media verification record is merged sequentially at each moment. In this case, the record at that moment is determined from the media verification record based on the event at that moment or the event in the event stream.

[0067] In the aforementioned backtracking method, the event stream of interface changes is acquired, and interface change result records are generated based on the event stream. This creates corresponding documents for the events and results caused by user operations. This process is implemented for interfaces with backtracking requirements, reducing resource consumption and helping to minimize user privacy leaks. When using event streams to generate corresponding interface change result records, it is not necessary to record the entire screen directly, and recording can be done at the moment the event stream occurs, allowing for efficient processing with fewer resources. When the interface changes, media records are acquired based on the browser's communication module, which can overcome the platform rule limitations of the system program, ensuring the integrity and stability of media data during interface changes, avoiding the cross-platform adaptation work required for application development, and improving the convenience and universality of the verification process. Finally, the interface change result records and media records are sent to the server. The server is used to merge the interface change result records and media records into a media verification record, enabling efficient processing of the corresponding data and preventing the recording process before backtracking from affecting normal user use. Finally, in response to the verification command, a backtracking process is performed based on the media verification records. This process combines static interface change results records with dynamic media verification records for comprehensive analysis, ensuring the accuracy of the backtracking.

[0068] In some embodiments, obtaining an event stream of interface changes and generating an interface change result record based on the event stream includes: obtaining an operation sequence triggered by the interface; generating an event stream of the interface based on the operation sequence and the interface change events triggered by the operation sequence; persisting the document object model corresponding to the event stream to obtain an interface change result record; wherein, the interface change result record includes an operation sequence arranged in chronological order and the interface change events triggered by the operation sequence.

[0069] An operation sequence is a series of user commands on the interface, arranged in a specific order. Operation sequences are a type of user event, and they can trigger changes in the interface, thus creating interface change events.

[0070] Interface change events represent the results of a sequence of operations. Each interface change event corresponds to one or more operation instructions. For example, the content represented by an interface change event could be the display of a corresponding pop-up window in response to a swipe or click instruction. The chronological order can be sequential, reverse chronological, or interpolated.

[0071] In one embodiment, an event stream is generated by listening to the operation sequence corresponding to user events and changes in the user interface DOM.

[0072] In one embodiment, the user's sequence of actions on the interface can be captured, a structured event log can be generated, and the action log can be persisted to support efficient querying.

[0073] In this embodiment, the user's operation sequence triggered on the interface and the result of the operation sequence on the interface are both regarded as events of the interface, thereby forming corresponding event relationships and constructing an event flow; then the document object model is persisted, so that the interface elements and corresponding events in the interface change result record exist in a tree-like form, which is convenient for backtracking in the form of code and realizing efficient query.

[0074] In one embodiment, such as Figure 3 As shown, the browser-based communication module acquires media recordings when the interface changes, including step 302; correspondingly, the interface change result record and media recordings are sent to the server, including step 304; wherein:

[0075] Step 302: When adjusting the interface of the application to be traced in the event stream triggered by the application to be traced, the media data collected by the device is obtained based on the browser's communication module.

[0076] The application to be traced is the application that needs to be recorded and traced. The application to be traced can be the same as or a different application from the browser. The application to be traced has corresponding user authorization so that the relevant media data is obtained with the user's permission. Media data is video or audio data obtained by the terminal device from outside the device, such as video data captured by a camera or audio data captured by a microphone.

[0077] In some embodiments, adjusting the interface of the application to be traced by the event flow triggered by the application to be traced means that multiple events are triggered by the user's operation on the interface of the application to be traced, and all multiple events are used to adjust the interface elements carried by the interface.

[0078] In some embodiments, the browser-based communication module acquires media data collected by the device, including: recording data collected by the terminal device through a camera or microphone using the browser-based communication module to obtain media data. Therefore, as long as the application being traced has authorization, the data collected by the terminal device and the application can be recorded again through the browser's communication module, eliminating the need for permission requests from the recording program and reducing the frequency of user operations.

[0079] Step 304: Send the interface change result record and media data to the server; the server is used to merge the interface change result record and media data into a media verification record.

[0080] In some embodiments, sending interface change results and media data to a server includes sending interface change results and media data to a server at the same time.

[0081] In some embodiments, sending interface change result records and media data to the server includes: sequentially sending interface change result records and media data at the same time to the server in chronological order.

[0082] In this embodiment, the recording steps prior to the backtracking process are related to the event stream triggered by a specific application on the interface. Data collection from this event stream ensures temporal synchronization between media data and the event stream, reducing media data redundancy, minimizing computational resources required, and guaranteeing backtracking effectiveness. Furthermore, since terminals such as Android and iOS may use different video encoding formats, the browser-based communication module-based data collection method can omit the encoding adaptation process, reducing computational resources required for encoding adaptation and ensuring processing efficiency. This minimizes the impact of the recording process on normal user experience.

[0083] In some embodiments, sending the interface change result record and media record to the server includes: converting media data into a media stream; sending the interface change result record and media stream to the server; wherein the server is used to store the media stream as a media file and record the interface change result record and media file as a media verification record; the metadata associated with the media stream is stored in a database, and the metadata includes the storage path of the media verification record.

[0084] Media streams are data transmitted and processed sequentially in the form of frames, packets, or blocks. When using media streams, the server does not need to wait for the entire media stream to be transmitted before recording can begin, ensuring processing efficiency. Media streams can be real-time captured images or audio.

[0085] A media file is a complete collection of data stored in file format. A media stream is dynamic and continuous data; when using a media stream, the acquired stream data can be processed directly. A media file, on the other hand, is a static and complete collection of data; when using a media file, it needs to be read and used based on the complete media file.

[0086] The metadata includes at least the storage path to identify the media verification records stored on the server. The metadata also includes data such as user ID and time. The user ID is the user identifier for the application being traced, while the time in the metadata is used to synchronize the interface change results with the media files to form the media verification record.

[0087] In one embodiment, the terminal captures media data and performs streaming processing on it; local or remote media data can be acquired via WebRTC and recorded as a media stream. The server then records the media stream as a file. The recorded file is then persistently stored, and metadata is recorded to provide a user interface for playback and to support online playback of the recorded audio and video files.

[0088] In one embodiment, when a user enters the interface, the user's basic information is obtained; data can then be associated with it. In addition, the media file can be a video, which is stored on a video server, and the metadata associated with the video is stored in a database.

[0089] In one embodiment, the metadata contains the storage path of the media file and acts as an index to the media data to facilitate the retrieval of the corresponding media file.

[0090] In this embodiment, the recorded media data is first converted into a media stream, and then the corresponding media stream is transmitted to the server for recording. This eliminates the need for local storage of media files, preventing local terminal tampering with the video and reducing transmission time during file storage, thus ensuring processing efficiency. When using server-side recording, it also reduces the need for adapting encoding formats for various types of terminal devices, eliminating the need to address format differences and the resulting file size issues. Furthermore, the metadata is associated with media verification records, allowing the terminal to use metadata as an index and locate the corresponding media verification record based on the storage path in the metadata. This ensures that the metadata and media verification record lookup processes are linked, guaranteeing efficient lookup.

[0091] In some embodiments, the server is used to create a virtual desktop and a virtual sound card. The media verification record includes a playback screen and playback audio, and the media record corresponds to a media file. The server is used to place the screen corresponding to the interface change result record and the screen contained in the media file on the virtual desktop, and record the virtual desktop to obtain the playback screen. The playback screen includes the screen of the media file at each moment, and the screen composed of the behavior content of each behavior position corresponding to each moment of the interface change result record. The server is used to record the audio corresponding to the media file through the virtual sound card to obtain the playback audio. The playback audio includes the audio of the media file at each moment.

[0092] A virtual desktop is a virtual device on a server that simulates a physical monitor; related screens can be arranged in a unified virtual desktop, and screen blending can be achieved directly through screen recording. A virtual sound card is a virtual device that simulates the functions of a hardware sound card. It simulates the behavior of a hardware sound card through a program, thereby processing audio data streams through the virtual interface of the virtual sound card.

[0093] The backtracking screen is a combination of the screen corresponding to the interface change result record and the screen contained in the media file. Since the backtracking screen is recorded by the server, the adaptation process between the screen corresponding to the interface change result record and the screen contained in the media file can be omitted, thus ensuring processing efficiency.

[0094] The interface change result record includes the action position and the action content at each moment. The corresponding screen can be reconstructed by the action position and the action content at that position. Here, the action position is the position where the user operates on the interface, and the action content at that position is the user's input or output at that position. This input or output can be text or interface elements that change based on events such as clicks. For example, if the user clicks at action position A at a certain moment, and a window is displayed at action position A at the next moment, the user click is the action, and the window displayed at action position A is the interface element that changes based on that action.

[0095] This can be achieved by adding highlights to the locations where the user triggers the screen or clicks the mouse, so that the playback screen shows the location of the user's operation. The behavior content can rely on log records, allowing input content to be recovered through user input behavior, and the corresponding user operation can be accurately reconstructed based on the log records.

[0096] The playback audio is recorded from the media file at each moment, matching the playback screen. The interface change results are recorded at the moment corresponding to each event, and the media file also has a corresponding moment for each event. Since the media file contains both video and audio, the start time of each operation is marked and then checked to avoid errors caused by timestamps, thus achieving synchronization between the playback screen and playback audio.

[0097] In one embodiment, data acquisition determines and accesses the web address of the collaborative playback engine through the storage path in the metadata. At this web address, the user's complete playback screen and playback audio are played through a virtual screen to reproduce the user's scene. At the same time, the scene is recorded using the ffmpeg tool to achieve fusion.

[0098] In one example, during the initialization phase, the virtual rendering process generates a virtual desktop and a virtual sound card, loads the target webpage, and configures audio routing rules (system audio capture range); during the data acquisition phase, the virtual rendering process is captured, and audio and video data are acquired in real time; finally, during the compositing and output phase, the virtual rendering frame sequence, video track encoding, and audio track encoding are acquired in real time and merged to generate a video file.

[0099] In this embodiment, since the server does not have a physical display screen and a physical sound card, a corresponding virtual desktop and a virtual sound card are created to accommodate various types of data. Recording the interface change results and the corresponding screens contained in the media files allows both screen combinations to be accommodated in the virtual desktop. Screen fusion can be achieved by recording the backtracking screen to obtain the backtracking screen. At the same time, backtracking audio is recorded according to time, so that the backtracking screen and backtracking audio are unified.

[0100] In some embodiments, the media verification record includes behavior type, replay video, and replay audio; different behavior types correspond to different markers for their operation events. Replaying based on the media verification record includes: displaying a timeline for replay, with multiple points on the timeline marked, each marker indicating the behavior type at each point; and playing the replay video and replay audio according to instructions given for the timeline.

[0101] The behavior category represents the content corresponding to the user's action; behavior categories include, but are not limited to, clicking, swiping up, and input, etc., and each category can be marked with different shapes. For example, the mark shape for clicking is a dot, and the mark shape for input is a label.

[0102] In one embodiment, playing backlit images and backlit audio according to instructions for the timeline includes: controlling the playback of backlit images and backlit audio according to play and pause instructions for the timeline.

[0103] The timeline commands can include instructions such as progress bar dragging and playback speed adjustment to ensure precise control. Playback of video and audio replays are historical event stream replays, dynamically rendering historical operations such as highlighting click locations and restoring input content to recreate the scene.

[0104] In this embodiment, in the case of rewinding, the timeline has markers corresponding to the type of behavior to convey the corresponding event information, so that users can extract and predict the moment that needs to be displayed through the timeline, and then select the rewinding screen and rewinding audio in a targeted manner to efficiently replay the recorded operation.

[0105] In one example, the recording of interface change results is a page behavior backtracking mechanism, and the server's communication module is based on the browser's native WebRTC communication framework. Therefore, this embodiment provides a real-time screen recording system that deeply integrates the browser's native WebRTC communication framework with the page behavior backtracking mechanism. Its core technological value lies in the following: Real-time acquisition and transmission of audio and video data on the browser side is achieved using the WebRTC standard protocol, breaking through platform limitations and naturally supporting multiple terminal devices such as Windows, macOS, iOS, and Android. Through the collaborative architecture of page behavior backtracking technology and the real-time audio and video acquisition module, a full-scene recording system covering static page operations and dynamic audio and video acquisition is constructed. A standardized Web technology development model is adopted, avoiding the cross-platform adaptation work required for APP development, significantly shortening the development cycle and reducing maintenance costs. Users do not need to install a dedicated application; high-quality real-time audio and video acquisition can be achieved through a browser, greatly improving the convenience and universality of the verification process.

[0106] This embodiment achieves a technological leap from "limited coverage" to "full platform support" and from "high-cost operation" to "lightweight deployment" through technological innovation, providing a reliable technical solution for application scenarios such as remote verification and online authentication. Among them, the real-time audio and video acquisition and transmission based on the browser's native WebRTC, and the collaborative integration of web page behavior backtracking and real-time audio and video acquisition, realize the standardization of platform compatibility and low development cost, and form a verification process guarantee with high stability and resistance to environmental interference.

[0107] In one embodiment, a traditional app screen recording solution, tested on iOS and Android systems, requires whitelist authentication of the application interface before screen recording can be performed when recording from a single application and an application with an embedded H5 page. Other pages cannot be accessed, and the authentication process is too frequent.

[0108] Correspondingly, when verifying the cross-platform screen recording function using steps 202-208 of this case and the corresponding embodiments, the test environment is as follows:

[0109] Browsers: Chrome 110 (Windows), Safari 15 (macOS), Chrome 105 (Android), Safari 16 (iOS).

[0110] Network conditions: 5G / 4G / Wi-Fi hybrid simulation.

[0111] Test duration: 10 minutes of continuous recording per platform.

[0112] Test scenario: Browser, application with embedded H5 page.

[0113] After processing using steps 202-208 and the corresponding embodiments in this case, when a test webpage containing a dynamic form is opened in various browsers, and permissions for media devices such as cameras and microphones are obtained, if operation instructions corresponding to text input or button clicks are detected, the browser's WebRTC recording is initiated, synchronously capturing page DOM changes and real-time audio and video streams. The server receives the streaming media data, transcodes it, and stores it. Therefore, all devices can record the operation process normally, with clear and smooth video. The video resolution of both computers and mobile phones reaches high-definition levels, and the audio-visual synchronization error is controlled within 0.3 seconds, making the delay almost imperceptible to the human ear. A 10-minute video file is approximately 30MB in size.

[0114] The comparison results for this embodiment are shown in Table 1, as follows:

[0115] Table 1

[0116]

[0117] In one example, the corresponding test results are as follows: Video encoding optimization: Using VP9 / SVC layered encoding, 1080p@30fps is achieved with 1Mbps bandwidth; compared with the H.264 scheme, the bitrate is reduced by 37% at the same image quality; the adaptive bitrate control algorithm for anti-network jitter capability keeps the smoothness ≥95% when the packet loss rate is ≤5%; compared with the unoptimized scheme, the number of stutters is reduced by 62%; Evidence compliance: the timestamp accuracy reaches ±5ms (NTP calibration), supports SHA-256 hash verification, and the completeness of the evidence chain meets the judicial appraisal standard.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides a backtracking device for implementing the backtracking method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more backtracking device embodiments provided below can be found in the limitations of the backtracking method described above, and will not be repeated here.

[0120] In one exemplary embodiment, such as Figure 4 As shown, a backtracking device is provided, comprising:

[0121] Interface module 402 is used to acquire the event stream of interface changes and generate interface change result records based on the event stream;

[0122] Media module 404 is used by the browser-based communication module to obtain media records when the interface changes;

[0123] The fusion module 406 is used to send the interface change result record and the media record to the server; the server is used to fuse the interface change result record and the media record into a media verification record;

[0124] The backtracking module 408 is used to perform backtracking based on the media verification record in response to the verification command.

[0125] In one embodiment, the media module 404 is configured to:

[0126] When the interface of the application to be traced is adjusted in the event stream triggered by the application to be traced, the media data collected by the device is obtained based on the browser's communication module;

[0127] The fusion module 406 is used for:

[0128] The interface change result record and the media data are sent to the server; the server is used to merge the interface change result record and the media data into a media verification record.

[0129] In one embodiment, the fusion module 406 is configured to:

[0130] Convert the media data into a media stream;

[0131] The interface change results are recorded and sent to the server along with the media stream;

[0132] The server is used to store the media stream as a media file and record the interface change results and the media file as a media verification record; the metadata associated with the media stream is stored in a database, and the metadata includes the storage path of the media verification record.

[0133] In one embodiment, the server is used to create a virtual desktop and a virtual sound card, and the media verification record includes a playback screen and a playback audio, the media record corresponding to a media file;

[0134] The server is used to place the screen corresponding to the interface change result record and the screen contained in the media file on the virtual desktop, and record the virtual desktop to obtain the retrospective screen; the retrospective screen includes the screen of the media file at each moment, and the screen composed of the behavior content of each behavior position corresponding to each moment of the interface change result record.

[0135] The server is used to record the audio corresponding to the media file through the virtual sound card to obtain the retrospective audio; the retrospective audio includes the audio of the media file at each of the stated times.

[0136] In one embodiment, the media verification record includes behavior type, playback screen, and playback audio; different operation events for different behavior types correspond to different tags;

[0137] The backtracking module 408 is used for:

[0138] A timeline for backtracking is displayed, with multiple points in the timeline marked, each mark representing the type of behavior at each point in time;

[0139] Play the flashback footage and the flashback audio according to the instructions for the timeline.

[0140] In one embodiment, the interface module 402 is used for:

[0141] Obtain the operation sequence triggered by the interface, and generate the event stream of the interface based on the operation sequence and the interface change events triggered by the operation sequence;

[0142] Persist the document object model corresponding to the event stream to obtain a record of the interface change results;

[0143] The interface change result record includes the operation sequence arranged in chronological order and the interface change events triggered by the operation sequence.

[0144] Each module in the aforementioned backtracking device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0145] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a backtracking method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0146] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0149] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A backtracking method, characterized in that, The method includes: Obtain the event stream of interface changes, and generate an interface change result record based on the event stream; The browser-based communication module acquires media recordings when the interface changes. The interface change result record and the media record are sent to the server; the server is used to merge the interface change result record and the media record into a media verification record; In response to the verification command, backtracking is performed based on the media verification records.

2. The method according to claim 1, characterized in that, The browser-based communication module acquires media records when the interface changes, including: When the interface of the application to be traced is adjusted in the event stream triggered by the application to be traced, the media data collected by the device is obtained based on the browser's communication module; The step of sending the interface change result record and the media record to the server includes: The interface change result record and the media data are sent to the server; the server is used to merge the interface change result record and the media data into a media verification record.

3. The method according to claim 2, characterized in that, The step of sending the interface change result record and the media record to the server includes: Convert the media data into a media stream; The interface change results are recorded and sent to the server along with the media stream; The server is used to store the media stream as a media file and record the interface change results and the media file as a media verification record; the metadata associated with the media stream is stored in a database, and the metadata includes the storage path of the media verification record.

4. The method according to claim 1, characterized in that, The server is used to create virtual desktops and virtual sound cards, and the media verification record includes playback screen and playback audio. The media record corresponds to a media file. The server is used to place the screen corresponding to the interface change result record and the screen contained in the media file on the virtual desktop, and record the virtual desktop to obtain the retrospective screen; the retrospective screen includes the screen of the media file at each moment, and the screen composed of the behavior content of each behavior position corresponding to each moment of the interface change result record. The server is used to record the audio corresponding to the media file through the virtual sound card to obtain the retrospective audio; the retrospective audio includes the audio of the media file at each of the stated times.

5. The method according to claim 1, characterized in that, The media verification record includes the type of behavior, the playback video, and the playback audio; different operation events for different types of behavior correspond to different markers; The backtracking based on the media verification records includes: A timeline for backtracking is displayed, with multiple points in the timeline marked, each mark representing the type of behavior at each point in time; Play the flashback footage and the flashback audio according to the instructions for the timeline.

6. The method according to claim 1, characterized in that, The process of acquiring the event stream of interface changes and generating interface change result records based on the event stream includes: Obtain the operation sequence triggered by the interface, and generate the event stream of the interface based on the operation sequence and the interface change events triggered by the operation sequence; Persist the document object model corresponding to the event stream to obtain a record of the interface change results; The interface change result record includes the operation sequence arranged in chronological order and the interface change events triggered by the operation sequence.

7. A backtracking device, characterized in that, The device includes: The interface module is used to acquire the event stream of interface changes and generate interface change result records based on the event stream. The media module is used to acquire media records when the interface changes, based on the browser's communication module. The fusion module is used to send the interface change result record and the media record to the server; the server is used to fuse the interface change result record and the media record into a media verification record; The backtracking module is used to perform backtracking based on the media verification record in response to the verification command.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images