Video playback method and device, equipment and medium

Abstract

The embodiment of the invention provides a video playback method and device, equipment and a medium, and is used for solving the problem that video playback cannot be accurately and effectively realized when a version is changed in the related technology. In the embodiment of the invention, the electronic equipment obtains a playback identifier carried in a playback request and a version number of a service system currently applied by a client; acquiring a screen recording file corresponding to the playback identifier; if the version number is different from the initial version number, playing the screen recording file; acquiring a video frame image of the screen recording file; and based on the ffmpeg, converting each acquired video frame image into a video in a coding format corresponding to the version number. Therefore, the video of the coding format corresponding to the current version number can be generated based on the ffmpeg, and video playback can be accurately and effectively realized when the version changes.

Description

Technical Field

[0001] The present application relates to the field of computer technology and applications, and in particular to a video playback method, device, equipment and medium. Background Art

[0002] In business systems, for various purposes, it is often necessary to record the screen, track the operation behavior on the web page, and provide specific data or video references to record the user's operation behavior and analyze and troubleshoot problems. For example, in the financial field, due to the requirements of regulatory authorities, the process of users purchasing financial products must be traceable, so that when a business dispute occurs between users and financial companies, there is a trace to follow. Common recording methods include screen recording based on Web Real-Time Communication (WebRTC). WebRTC can completely record the entire display screen, but it cannot be done without user perception. The user needs to agree and select the recorded screen content; at the same time, the size of the recorded content is proportional to the recording time, which occupies a large storage space; after recording, it needs to be uploaded to the server, which affects system stability and user experience; when playing back, you need to find a suitable player by yourself, and the playback clarity will be lost.

[0003] In order to solve this problem, the related art proposes to record and replay the web (rrweb) for video recording and playback. rrweb has a wide range of application scenarios, including but not limited to product iteration: by reproducing user interactions, the product team can more intuitively understand user needs and optimize product design; user behavior analysis: help the market and operation teams to deeply explore user behavior patterns and formulate more accurate marketing strategies; problem troubleshooting: when users encounter problems, by reproducing their interaction process, the technical support team can quickly locate and solve the problem. However, the current rrweb only supports playback of screen recordings in the iframe sandbox environment, and cannot convert the json string of the video into a video in a specified format, such as mp4, avi, etc. In the field of financial business, when a business dispute occurs, users may request to view the video of the operation, and the regulatory authorities will also require the submission of video data. If the rrweb screen recording data always exists in text form, it may be detrimental to business processing and data handover. Moreover, if there is a version change problem such as version upgrade, it may not support the playback of the preset format of the playback video, and thus the video cannot be accurately and effectively played back. Summary of the invention

[0004] The embodiments of the present application provide a video playback method, apparatus, device and medium, which are used to solve the problem in the related art that video playback cannot be accurately and effectively implemented when the version changes.

[0005] In a first aspect, an embodiment of the present application provides a video playback method, the method comprising:

[0006] Receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; obtain a screen recording file corresponding to the playback identifier;

[0007] If the version number is different from the initial version number saved for the playback identifier, playing the screen recording file;

[0008] Obtain the video frame image of the screen recording file; based on the Fast Forward Picture Experts Group (ffmpeg), convert each obtained video frame image into a video in the encoding format corresponding to the version number, and send the video to the client so that the client can play back the video.

[0009] In a second aspect, an embodiment of the present application further provides a video playback device, the device comprising:

[0010] A receiving and obtaining module, configured to receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; and obtain a screen recording file corresponding to the playback identifier;

[0011] A playing module, configured to play the screen recording file if the version number is different from the initial version number saved for the playback identifier;

[0012] A processing module is used to obtain the video frame image of the screen recording file; based on ffmpeg, each obtained video frame image is converted into a video in the encoding format corresponding to the version number, and the video is sent to the client so that the client can play back the video.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0014] A memory for storing program instructions;

[0015] The processor is used to call the program instructions stored in the memory and execute the steps included in any one of the above-mentioned video playback methods according to the obtained program instructions.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for video playback as described in any one of the above items.

[0017] In the embodiment of the present application, the electronic device obtains the playback identifier carried in the playback request and the version number of the business system currently applied by the client; obtains the screen recording file corresponding to the playback identifier; if the version number is different from the initial version number, the screen recording file is played; obtains the video frame image of the screen recording file; based on ffmpeg, each obtained video frame image is converted into a video in the encoding format corresponding to the version number. Thus, based on ffmpeg, a video in the encoding format corresponding to the current version number can be generated, and then when the version changes, video playback can be accurately and effectively achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a video playback method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the working principle of rrweb provided in an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the process of rrweb recording and playback in the related technology;

[0022] Figure 4 A schematic diagram of a detailed process of video playback provided in an embodiment of the present application;

[0023] Figure 5 A structural diagram of a video playback device provided in an embodiment of the present application;

[0024] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0026] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0027] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0028] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0029] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0030] In order to solve the problem in the related art that video playback cannot be accurately and effectively achieved when the version changes, the embodiments of the present application provide a video playback method, device, equipment and medium.

[0031] The video playback method includes: an electronic device receives a playback request, obtains a playback identifier carried in the playback request, and a version number of a business system currently applied by a client that sends the playback request; obtains a screen recording file corresponding to the playback identifier; if the version number is different from an initial version number saved for the playback identifier, plays the screen recording file; obtains a video frame image of the screen recording file; based on ffmpeg, converts each obtained video frame image into a video in an encoding format corresponding to the version number, and sends the video to the client that sends the playback request, so that the client can play back the video.

[0032] Embodiment 1:

[0033] Figure 1 A schematic diagram of a video playback method provided in an embodiment of the present application, the process includes the following steps:

[0034] S101: Receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; and obtain a screen recording file corresponding to the playback identifier.

[0035] The video playback method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server, wherein the server may be a node.js server.

[0036] When the user has a playback requirement, he can select the playback identifier on the preset page of the client and click the preset button, for example, he can click the "Playback" button. At this time, the client can obtain the version number of the business system of the current application, and send the version number and the playback identifier to the electronic device in the playback request. It should be noted that the playback identifier can include the information of the account that the user logs in to the business system and the information of the specific business handled by the user, so that the playback identifier is unique and can accurately locate the specific screen recording file.

[0037] When the electronic device receives a playback request, it can obtain the playback identifier carried in the playback request and the version number of the business system. In order to achieve video playback, the electronic device can obtain the screen recording file corresponding to the playback identifier. In a possible implementation, the electronic device can obtain the screen recording file saved corresponding to the playback identifier based on the correspondence between the pre-saved identifier and the file. The screen recording file records the screen activities during the business handling corresponding to the previous playback identifier, which may include the user's operations, the response of the business system, and related audio and video content.

[0038] For example, in the insurance business, the electronic device can use the playback identifier to record a user's insurance purchase process. At this time, the playback identifier contains the insurance policy ID.

[0039] S102: If the version number is different from the initial version number saved for the playback identifier, play the screen recording file.

[0040] In order to accurately and effectively realize video playback, the electronic device can obtain the initial version number saved for the playback identifier, wherein the initial version number is the version number of the client's business system when recording the screen. And determine whether the version number carried in the playback request is different from the initial version number. If the version number carried in the playback request is different from the initial version number, the screen recording file needs to be format converted. Specifically, the electronic device can play the screen recording file.

[0041] S103: Obtain the video frame image of the screen recording file; based on ffmpeg, convert each obtained video frame image into a video in the encoding format corresponding to the version number, and send the video to the client so that the client can play back the video.

[0042] The electronic device can obtain the video frame image of the played screen recording file, and obtain the encoding format corresponding to the current version number of the client's business system, and convert each obtained video frame image into a video in the encoding format based on ffmpeg.

[0043] Among them, ffmpeg is a multimedia support library that implements almost all common data encapsulation formats, multimedia transmission protocols, and audio and video codecs, and provides a complete solution for recording, converting, and streaming audio and video. In the embodiment of this application, ffmpeg is needed to help implement the image-to-video operation.

[0044] After acquiring the video in the encoding format, the electronic device can send the acquired video to the client. After receiving the video, the client can play back the video, and the user can review past operations through the video.

[0045] It should be noted that SAAS (Software as a Service) products such as LogRocket are often used to implement video playback functions, and they can provide pixel-level accurate recording and playback services. However, since such products are usually deployed in the cloud, the customer's intranet environment is often unable to be directly connected due to network isolation and other reasons, thereby limiting its scope of use. The method provided in the embodiment of the present application not only supports the storage of text data of multiple versions of videos, effectively avoiding data conflict problems that may arise during the upgrade of the business system, but also ensures the clarity and accuracy of the video playback function. Moreover, the method does not need to rely on cloud services, so it can be normally applied to the customer's intranet environment to meet more usage requirements.

[0046] In the embodiment of the present application, the electronic device obtains the playback identifier carried in the playback request and the version number of the business system currently applied by the client; obtains the screen recording file corresponding to the playback identifier; if the version number is different from the initial version number, the screen recording file is played; obtains the video frame image of the screen recording file; based on ffmpeg, each obtained video frame image is converted into a video in the encoding format corresponding to the version number. Thus, based on ffmpeg, a video in the encoding format corresponding to the current version number can be generated, and then when the version changes, video playback can be accurately and effectively achieved.

[0047] Embodiment 2:

[0048] In order to accurately and effectively play back the video, based on the above embodiments, in the embodiment of the present application, playing the screen recording file includes:

[0049] Start a headless browser and create a Tab page based on the headless browser;

[0050] Play the screen recording file in the Tab page.

[0051] In order to accurately and effectively play back the video, the electronic device can start a headless browser. Specifically, the electronic device can run the Puppeteer program. Puppeteer starts a headless browser in the electronic device, creates a new Tab page based on the started headless browser, and plays the screen recording file on the created Tab page.

[0052] It should be noted that in order to obtain the screen recording file, the screen recording file is loaded in the Tab page of the headless browser. In the new Tab page, in order to ensure that the screen recording file has been fully loaded and can be played, the electronic device will listen to the onload event of the Tab page. When the onload event is triggered, it means that the page has been loaded. At this time, the electronic device will pass the screen recording file to the Tab page, and the player on the page will play it.

[0053] Embodiment 3:

[0054] In order to accurately and effectively perform video playback, based on the above embodiments, in the embodiment of the present application, the step of obtaining the video frame image of the screen recording file includes:

[0055] The video frame images in the screen recording file are obtained according to the preset frame interval.

[0056] In actual scenarios, screen recording files contain a large amount of data and take up a significant amount of storage space. If the video is regenerated directly based on each frame image in the screen recording, the newly generated video will also take up a lot of space. When transmitted to the client, it may not only affect the stability of the business system, but also significantly reduce the user experience. Based on this, the electronic device can obtain a small amount of video frame images in the screen recording file.

[0057] Specifically, since adjacent video frames often record the same operation content, the electronic device can set a preset frame interval and select video frame images of the screen recording file based on the frame interval. This ensures that each selected frame effectively captures the user's operation, thereby reducing the number of video frames while still maintaining the integrity of the operation process, thereby effectively reducing the storage space occupied by the generated video.

[0058] Taking the electronic device as a node.js server as an example, in the embodiment of the present application, the problem of video format conversion is solved by node.js+pupteer+ffmpeg. Puppeteer and ffmpeg programs are run through the nodejs server. Puppeteer starts a headless browser in the server and creates a new tab page; the new page listens to the onload event, passes the queried video data to the page and plays it; then Puppeteer starts to take screenshots according to the preset number of frames, obtains the video frame image, and saves the video frame image to the memory; when the screen recording file is finished playing, ffmpeg converts the video frame image into a video according to the specified encoding format.

[0059] Embodiment 4:

[0060] In order to obtain the screen recording file, based on the above embodiments, in the embodiment of the present application, the step of obtaining the screen recording file corresponding to the playback identifier includes:

[0061] Obtain a full snapshot of the document object model (DOM) tree of the page saved corresponding to the playback identifier and each DOM operation on the page;

[0062] Based on the record and replay web page rrweb, the full snapshot and each DOM operation are replayed;

[0063] Based on Puppeteer recording and playback content, a screen recording file is obtained.

[0064] Existing web page recording and playback technologies may not be able to accurately capture and reproduce all operations of the user on the page, and may result in the inability to accurately restore the original state of the page during playback. Based on this, in an embodiment of the present application, the electronic device obtains a screen recording file based on a full snapshot of the DOM tree of the page and each DOM operation on the page.

[0065] Specifically, the electronic device can obtain a full snapshot of the page DOM tree saved corresponding to the playback identifier and each DOM operation on the page. It should be noted that when recording, the client obtains a full snapshot of the DOM tree of the current page at the beginning of recording through specific tools or libraries, such as rrweb. This snapshot contains all elements, attributes and style information of the page, and is the initial state during playback. When the user interacts with the page, each DOM operation is recorded in real time. These operations may include adding, deleting, and modifying elements. By recording these operations, all user behaviors on the page can be completely restored. And save the identifier and the obtained full snapshot of the page DOM tree and each DOM operation on the page. And the client sends the obtained full snapshot of the page DOM tree and each DOM operation on the page to the electronic device.

[0066] In a possible implementation, the electronic device may locally store a full snapshot of the DOM tree of the page and a json string of each DOM operation on the page. When recording, the client sends the json string to the electronic device. When the electronic device needs to replay, it generates a full snapshot of the DOM tree of the page and each DOM operation on the page based on the json string. Specifically, how to generate a full snapshot of the DOM tree of the page and each DOM operation on the page based on the json string is a prior art and will not be repeated here.

[0067] After obtaining the full snapshot and each DOM operation, the electronic device replays the full snapshot and each DOM operation based on rrweb. Among them, rrweb serializes the page data and events at different time points into json strings by recording the snapshots of the page DOM tree and its operation changes; when the video needs to be replayed, the json string is reconstructed into a DOM tree to replay the web page data at the historical moment. Different from the traditional screen recording method, rrweb is based on DOM recording, does not need to rely on the API provided by the browser, does not need to obtain browser authorization, supports multi-channel recording on PC and mobile phones, has small recording capacity, and low performance overhead.

[0068] And based on Puppeteer, record the playback content and generate a screen recording file. During the playback process, you can use Puppeteer's screen recording function to generate a screen recording file. Among them, Puppeteer is a Node library that provides a high-level application programming interface (Application Programming Interface, API) to control Google Chrome through the developer tool (Chrome DevTools, DevTools) protocol, allowing Chrome to execute commands like a robot, including clicking, filling out forms, taking screenshots, generating PDFs, etc.; Puppeteer runs in headless mode by default, that is, no interface mode, and can generate a headless browser to support video playback.

[0069] In the embodiment of the present application, the electronic device can accurately capture and reproduce all the user's behaviors on the web page by obtaining a full snapshot of the DOM tree and recording each DOM operation, thereby ensuring the accuracy of the playback.

[0070] In the embodiment of the present application, the client is based on the rrweb framework to achieve pluggable access to the business system, decoupled from the business system, and realize web page recording and playback.

[0071] The embodiment of the present application is equivalent to providing a method for recording and replaying web pages based on rrweb. The embodiment of the present application implements video format conversion based on Puppeteer and ffmpeg, converts text data into videos of a specified format, solves the problem of only being able to replay videos online, and proposes a more effective screen recording and playback solution based on rrweb, realizes concise and efficient data storage, and supports playback of multiple versions of video data and generation of videos of a specified format.

[0072] Figure 2 A schematic diagram of the working principle of rrweb provided in an embodiment of the present application.

[0073] Depend on Figure 2 As you can see, we took a snapshot when the page was in the initial state (state 0). Then, the user's first operation (operation 1) caused the page to change from state 0 to state 1. Then, the user's second operation further caused the page to change from state 1 to state 2. These two operations, operation 1 and operation 2, can be collectively referred to as operations log (Oplog for short).

[0074] rrweb is an open source JavaScript library for recording and replaying web pages. It is mainly used to record and replay user operations on web pages, including DOM changes (node ​​creation, destruction), node attribute changes, text changes, mouse interactions, page element scrolling, window size changes, input, and mouse movement (specifically the visual position of the mouse). The technical implementation of rrweb mainly relies on the MutationObserver API, which is an interface provided by the browser for monitoring DOM changes. Through MutationObserver, rrweb can capture every change of the DOM on the page in real time, including attribute modifications, increase and decrease of child nodes, etc. In this way, we only need to make a complete DOM snapshot at the beginning of recording, and then record all operation data. These operation data are called Oplog. Through the snapshot + Oplog method, all changes to the page can be recorded. During the recording process, rrweb uses an efficient serialization algorithm to convert DOM changes into a data format that can be stored and transmitted. These data can be stored locally or sent to the server for persistent storage. When it is necessary to replay user operations, rrweb will gradually restore DOM changes in the sandbox environment based on these snapshot data, thereby realizing the playback of user operations. rrweb also provides a wealth of configuration options, such as playback speed, playback DOM container, etc., to meet the needs of different scenarios. In addition, rrweb also supports pause, double-speed playback, drag timeline and other functions, making the playback process more flexible and convenient.

[0075] Embodiment 5:

[0076] In order to accurately and effectively perform video playback, based on the above embodiments, in the embodiment of the present application, the playback of the full snapshot and each DOM operation based on rrweb includes:

[0077] According to the timestamps saved corresponding to each DOM operation, obtaining a first timestamp corresponding to a first preset DOM operation and a second timestamp corresponding to a second preset DOM operation in each DOM operation; and determining each valid DOM operation in each DOM operation whose corresponding occurrence time is between the first timestamp and the second timestamp;

[0078] Based on rrweb, the full snapshot and each valid DOM operation are replayed.

[0079] There are many problems with the current rrweb solution, such as the inability to determine the user's effective operations, which may lead to a large amount of junk data and increase transmission and storage costs; that is, in actual application scenarios, when users operate electronic devices, they often perform many unnecessary operations, such as checking information before starting specific business operations, and performing certain operations after the business operations are completed. These operations have no substantial value for subsequent playback or recording. Based on this, in order to improve playback efficiency and storage space utilization, electronic devices can identify and only save DOM operations that have a substantial impact on user behavior or page status, that is, valid DOM operations. Combine full snapshot technology with the recording of valid DOM operations to achieve more accurate and efficient playback functions.

[0080] Specifically, the electronic device can first record all DOM operations, and identify and record two key timestamps: a first timestamp corresponding to the first preset DOM operation and a second timestamp corresponding to the second preset DOM operation. These two timestamps define the time range of the DOM operations that need to be replayed. Among them, the first preset DOM operation can be a DOM operation for starting business processing, and the second preset DOM operation can be a DOM operation for ending business processing. According to the recorded timestamps, the electronic device filters out all valid DOM operations that occur between the first timestamp and the second timestamp. The electronic device takes the full snapshot (that is, the complete DOM state of the page at a certain moment) and the filtered valid DOM operations as input, and uses rrweb for playback. The playback process replays these operations in chronological order, thereby reproducing the user's interaction process within a specific time period.

[0081] Among them, the full snapshot of the DOM tree and the JSON string of each DOM operation can be called the text data of the video.

[0082] In a business system, the user's operation behavior generally has a beginning and an end. From the moment the user enters the business system, it is necessary to clarify the user's operation purpose in order to determine the start time of rrweb recording; during the user's operation, we need to clarify the user's effective operation; when the user's operation is finished, it is necessary to clarify the end time of the user's operation, so as to stop rrweb recording. If in the entire process, the above matters cannot be determined and each DOM operation is determined as a valid operation, it will increase storage pressure and reduce the stability of the business system. The method provided in the embodiment of the present application is used to analyze the user's effective operations, dynamically store the text data of the video, eliminate redundant data and dirty data, and realize concise and efficient data storage. By recording only valid DOM operations, the amount of data that needs to be processed during playback can be significantly reduced, and the playback speed and user experience can be improved. Moreover, saving only valid operations can reduce the occupation of storage resources, especially in long-term or high-frequency user interaction scenarios, this optimization is particularly important.

[0083] It should be noted that rrweb is a powerful library, which mainly consists of three core packages: rrweb, rrweb-snapshot and rrweb-player. These three packages work together to provide users with a complete page recording and playback solution. rrweb mainly provides two methods: record and replay. Record is responsible for recording the full amount of DOM information from the page loading, and continuously monitoring the changes of the page (i.e. DOM mutations). Whenever the page changes, this method will emit the change information in JSON format for developers to use; replay is responsible for reorganizing a series of JSON data recorded by record and then replaying the page content at that time. rrweb-snapshot mainly provides two methods used in record: serializeNodeWithId for serializing node nodes to obtain the change information and snapshot for obtaining page snapshots; in addition, it also provides a method used in replay: restore the page snapshot to help build the rebuild of the playback DOM. rrweb-player has designed a set of new UI players for rrweb, which can realize functions such as dragging the progress bar and adjusting the playback speed.

[0084] The business system implements the page recording and playback functions through API calls between the three packages of rrweb. The whole process is divided into two stages: recording and playback. In the recording stage, rrweb first completes a full snapshot of the page DOM tree, and then needs to monitor all events that may cause changes to the view based on the current view state through MutationObserver, such as: DOM changes (node ​​creation, destruction, node attribute changes, text changes), mouse movement, mouse interaction, page or element scrolling, window size changes, input; the whole monitoring process is similar to git submitting a version first and then appending it each time. After monitoring the relevant DOM tree and changed event data, the data is serialized into an easy-to-read json data format, and then the json data format is sent to the backend server through interface calls and other methods. The backend server calls the corresponding database for data storage. In the playback stage, the front-end system calls the back-end server to query the video's json data. After rrweb gets the data, in order to block the execution of unknown scripts in the json data, it will replay the video in an iframe sandbox environment.

[0085] Figure 3 It is a schematic diagram of the rrweb recording and playback process in the related art.

[0086] Depend on Figure 3It can be seen that during the recording process, the client obtains various DOM operations, records and serializes the DOM operations, and the electronic device stores the data in the database. During the playback process, the electronic device queries the data from the database and sends the data to the client. The client reads the data and plays it based on the rrweb player.

[0087] Figure 4 A schematic diagram of the detailed process of video playback provided in an embodiment of the present application.

[0088] Depend on Figure 4 It can be seen that during the recording process, the client obtains various DOM operations, records and serializes the events of the DOM operations, and performs timed storage. The electronic device stores the data in the database through timed storage, and records the corresponding initial version number. In addition, the electronic device periodically clears dirty data, that is, the data whose time interval of storage exceeds the preset time interval described in the embodiment of the present application; during the playback process, the electronic device queries the data from the database, and sorts the data in ascending time order, sends the data to the client, and the client reads the data. Moreover, during the video conversion process, the electronic device starts a headless browser based on Puppteer, creates a new Tab page, triggers the onload event, plays the screen recording file generated corresponding to the queried data, and obtains the video frame image, converts the obtained video frame image into a video based on ffmpeg, and replaces the version. The client uses the rrweb player for playback. At this time, a multi-version system is implemented based on nginx / cdn.

[0089] Embodiment 6:

[0090] In order to accurately and effectively playback, based on the above embodiments, in the embodiment of the present application, after obtaining the video frame image of the screen recording file, before converting each obtained video frame image into a video in the encoding format corresponding to the version number based on ffmpeg, the method further includes:

[0091] If any video frame image contains a domain name or a hyperlink, the initial version number saved for the playback identifier is added to the preset position of the domain name or the hyperlink.

[0092] If the client-side business system has a version iteration, the image storage address or hyperlink may become invalid, so that the corresponding address cannot be found during video playback, resulting in white screen, distorted screen and other problems. In order to effectively deal with this problem, electronic devices have adopted the method of embedding the version number into the domain name or hyperlink to ensure the long-term validity of the link.

[0093] Specifically, when the electronic device captures a video frame image containing a domain name or a hyperlink, it will automatically embed the initial version number in a preset position of the domain name or the hyperlink, such as immediately after the last character of the domain name or the hyperlink. Using the method provided in the embodiment of the present application, the user can accurately access the resources before the update through the domain name or the hyperlink with the version number, or be redirected to the latest valid resource when necessary.

[0094] The video json strings recorded by rrweb come from different moments. The front-end pages at that time may be in different versions due to the upgrade of the business system, resulting in path conflicts in the stored data such as picture links, so the problem of multiple versions of data must be handled. In the embodiment of the present application, the path conflict problem is solved by multi-version recording. In the recording stage, each time the data is stored, the version number of the current business system is marked, that is, the initial version number described in the embodiment of the present application, and the json string of the video is stored together with the version number and the playback identifier; in the playback stage, after the data is queried, all domain names or the hyperlinks are brought with the version number of the business system through path replacement; at the same time, all historical versions of the business system are saved through components such as nginx or cdn. When the domain name or the domain name and version number of the hyperlink are changed, it can point to a specific version of the data. If the user's recording data is in different business versions, cross-version addressing will be performed when the video is played back.

[0095] It should be noted that during the recording process, rrweb usually performs some special processing when serializing the full snapshot of the DOM tree and DOM operations, such as de-scripting. All JavaScript on the recorded page should not be executed. When recording, you can simply record a tag value without recording all the possible script contents or the view state that is not reflected in the HTML. For example, <input> The input value will not be reflected in its HTML, but will be recorded through the value attribute and the relative path will be converted to an absolute path, so as to avoid reading the relative path during playback and record the content of the Cascading Style Sheets (CSS) as much as possible. Among them, the parsed CSS rules are obtained, and all the styles that can be obtained are inlined during recording, so that the recording of some intranet environments can also have better effects. However, when the relative path is converted to an absolute path, the corresponding address cannot be found during video playback, resulting in problems such as a white screen and a distorted screen. The method provided in the embodiment of the present application can overcome this problem.

[0096] In order to accurately and effectively perform video playback, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0097] Obtain target timestamps corresponding to a preset number of DOM operations in each DOM operation, and when a time interval between the target timestamp and a currently corresponding timestamp exceeds a preset interval, delete the full snapshot and each DOM operation.

[0098] Since in actual scenarios, the full snapshot corresponding to the playback identifier and each DOM operation usually do not need to be replayed after a long time, in order to save resources, the electronic device can adopt the following strategy: it first records the target timestamp of the preset number of operations in each DOM operation. Then, when the time interval between this target timestamp and the current timestamp exceeds the preset threshold, the electronic device automatically deletes the relevant full snapshot and each relevant DOM operation.

[0099] The embodiment of the present application is based on the rrweb framework, and realizes more effective page recording and playback functions by optimizing the text data transmission and storage process of the video, realizing multi-version screen recording, and supporting rich video format conversion, thereby improving the practical application capability of rrweb.

[0100] Embodiment 7:

[0101] Based on the same inventive concept, the present application embodiment also provides a video playback device, please refer to Figure 5 , the device comprises:

[0102] The receiving and obtaining module 501 is used to receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; and obtain a screen recording file corresponding to the playback identifier;

[0103] A playing module 502, configured to play the screen recording file if the version number is different from the initial version number saved for the playback identifier;

[0104] The processing module 503 is used to obtain the video frame image of the screen recording file; based on ffmpeg, each obtained video frame image is converted into a video in the encoding format corresponding to the version number, and the video is sent to the client so that the client can play back the video.

[0105] In a possible implementation, the playing module 502 is specifically used to start a headless browser, create a Tab page based on the headless browser, and play the screen recording file in the Tab page.

[0106] In a possible implementation, the processing module 503 is specifically configured to obtain the video frame images in the screen recording file according to a preset frame interval.

[0107] In a possible implementation, the receiving and obtaining module 501 is specifically used to obtain a full snapshot of the DOM tree saved corresponding to the playback identifier and each DOM operation on the page; based on rrweb, the full snapshot and each DOM operation are played back; based on Puppeteer recording and playback content, a screen recording file is obtained.

[0108] In a possible implementation, the receiving and obtaining module 501 is specifically configured to obtain, according to the timestamp saved corresponding to each DOM operation, a first timestamp corresponding to a first preset DOM operation and a second timestamp corresponding to a second preset DOM operation in each DOM operation; and determine each valid DOM operation whose corresponding occurrence time in each DOM operation is between the first timestamp and the second timestamp; based on rrweb, replay the full snapshot and each valid DOM operation.

[0109] In a possible implementation, the processing module 503 is further configured to add an initial version number stored for the playback identifier at a preset position of the domain name or the hyperlink if any video frame image contains a domain name or a hyperlink.

[0110] In a possible implementation, the processing module 503 is also used to obtain the target timestamp corresponding to a preset number of DOM operations in each DOM operation, and to delete the full snapshot and each DOM operation when the time interval between the target timestamp and the current corresponding timestamp exceeds a preset interval.

[0111] Embodiment 8:

[0112] Based on the same inventive concept, the present application embodiment provides an electronic device, which can realize the function of the video playback mentioned above, please refer to Figure 6 The device includes a processor 601, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604.

[0113] The memory 603 stores a computer program. When the program is executed by the processor 601, the processor 601 performs the following steps:

[0114] Receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; obtain a screen recording file corresponding to the playback identifier;

[0115] If the version number is different from the initial version number saved for the playback identifier, playing the screen recording file;

[0116] Obtain the video frame image of the screen recording file; based on ffmpeg, convert each obtained video frame image into a video in the encoding format corresponding to the version number, and send the video to the client so that the client can play back the video.

[0117] In a possible implementation, playing the screen recording file includes:

[0118] Start a headless browser and create a Tab page based on the headless browser;

[0119] Play the screen recording file in the Tab page.

[0120] In a possible implementation, obtaining the video frame image of the screen recording file includes:

[0121] The video frame images in the screen recording file are obtained according to the preset frame interval.

[0122] In a possible implementation, obtaining the screen recording file corresponding to the playback identifier includes:

[0123] Obtain a full snapshot of the page document object model DOM tree saved corresponding to the playback identifier and each DOM operation on the page;

[0124] Based on rrweb, replay the full snapshot and each DOM operation;

[0125] Based on Puppeteer recording and playback content, a screen recording file is obtained.

[0126] In a possible implementation, the replaying of the full snapshot and each DOM operation based on rrweb includes:

[0127] According to the timestamps saved corresponding to each DOM operation, obtaining a first timestamp corresponding to a first preset DOM operation and a second timestamp corresponding to a second preset DOM operation in each DOM operation; and determining each valid DOM operation in each DOM operation whose corresponding occurrence time is between the first timestamp and the second timestamp;

[0128] Based on rrweb, the full snapshot and each valid DOM operation are replayed.

[0129] In a possible implementation, after obtaining the video frame image of the screen recording file, and before converting each obtained video frame image into a video in a coding format corresponding to the version number based on ffmpeg, the method further includes:

[0130] If any video frame image contains a domain name or a hyperlink, the initial version number saved for the playback identifier is added to the preset position of the domain name or the hyperlink.

[0131] In a possible implementation, the method further includes:

[0132] Obtain target timestamps corresponding to a preset number of DOM operations in each DOM operation, and when a time interval between the target timestamp and a currently corresponding timestamp exceeds a preset interval, delete the full snapshot and each DOM operation.

[0133] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0134] The communication interface 602 is used for communication between the electronic device and other devices.

[0135] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0136] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0137] Embodiment 9:

[0138] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0139] Receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; obtain a screen recording file corresponding to the playback identifier;

[0140] If the version number is different from the initial version number saved for the playback identifier, playing the screen recording file;

[0141] Obtain the video frame image of the screen recording file; based on ffmpeg, convert each obtained video frame image into a video in the encoding format corresponding to the version number, and send the video to the client so that the client can play back the video.

[0142] In a possible implementation, playing the screen recording file includes:

[0143] Start a headless browser and create a Tab page based on the headless browser;

[0144] Play the screen recording file in the Tab page.

[0145] In a possible implementation, obtaining the video frame image of the screen recording file includes:

[0146] The video frame images in the screen recording file are obtained according to the preset frame interval.

[0147] In a possible implementation, obtaining the screen recording file corresponding to the playback identifier includes:

[0148] Obtain a full snapshot of the page document object model DOM tree saved corresponding to the playback identifier and each DOM operation on the page;

[0149] Based on rrweb, replay the full snapshot and each DOM operation;

[0150] Based on Puppeteer recording and playback content, a screen recording file is obtained.

[0151] In a possible implementation, the replaying of the full snapshot and each DOM operation based on rrweb includes:

[0152] According to the timestamps saved corresponding to each DOM operation, obtaining a first timestamp corresponding to a first preset DOM operation and a second timestamp corresponding to a second preset DOM operation in each DOM operation; and determining each valid DOM operation in each DOM operation whose corresponding occurrence time is between the first timestamp and the second timestamp;

[0153] Based on rrweb, the full snapshot and each valid DOM operation are replayed.

[0154] In a possible implementation, after obtaining the video frame image of the screen recording file, and before converting each obtained video frame image into a video in a coding format corresponding to the version number based on ffmpeg, the method further includes:

[0155] If any video frame image contains a domain name or a hyperlink, the initial version number saved for the playback identifier is added to the preset position of the domain name or the hyperlink.

[0156] In a possible implementation, the method further includes:

[0157] Obtain target timestamps corresponding to a preset number of DOM operations in each DOM operation, and when a time interval between the target timestamp and a currently corresponding timestamp exceeds a preset interval, delete the full snapshot and each DOM operation.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0162] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video playback method, characterized in that: The method comprises: Receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; obtain a screen recording file corresponding to the playback identifier; If the version number is different from the initial version number saved for the playback identifier, playing the screen recording file; Obtain video frame images of the screen recording file; based on the fast forward picture experts group ffmpeg, convert each acquired video frame image into a video in the encoding format corresponding to the version number, and send the video to the client so that the client can play back the video.

2. The method according to claim 1, characterized in that Playing the screen recording file includes: Start a headless browser and create a Tab page based on the headless browser; Play the screen recording file in the Tab page.

3. The method according to claim 1, characterized in that The step of obtaining the video frame image of the screen recording file includes: The video frame images in the screen recording file are obtained according to the preset frame interval.

4. The method according to claim 1, characterized in that: The obtaining of the screen recording file corresponding to the playback identifier includes: Obtain a full snapshot of the page document object model DOM tree saved corresponding to the playback identifier and each DOM operation on the page; Based on the record and replay web page rrweb, the full snapshot and each DOM operation are replayed; Based on Puppeteer recording and playback content, a screen recording file is obtained.

5. The method according to claim 4, characterized in that The replay of the full snapshot and each DOM operation based on rrweb includes: According to the timestamps saved corresponding to each DOM operation, obtaining a first timestamp corresponding to a first preset DOM operation and a second timestamp corresponding to a second preset DOM operation in each DOM operation; and determining each valid DOM operation in each DOM operation whose corresponding occurrence time is between the first timestamp and the second timestamp; Based on rrweb, the full snapshot and each valid DOM operation are replayed.

6. The method according to any one of claims 1 to 5, characterized in that: After obtaining the video frame image of the screen recording file, and before converting each obtained video frame image into a video in a coding format corresponding to the version number based on ffmpeg, the method further includes: If any video frame image contains a domain name or a hyperlink, the initial version number saved for the playback identifier is added to the preset position of the domain name or the hyperlink.

7. The method according to claim 5, characterized in that The method further comprises: Obtain target timestamps corresponding to a preset number of DOM operations in each DOM operation, and when a time interval between the target timestamp and a currently corresponding timestamp exceeds a preset interval, delete the full snapshot and each DOM operation.

8. A video playback device, characterized in that: The device comprises: A receiving and obtaining module, configured to receive a playback request, obtain a playback identifier carried in the playback request, and a version number of a business system currently applied by the client sending the playback request; and obtain a screen recording file corresponding to the playback identifier; A playing module, configured to play the screen recording file if the version number is different from the initial version number saved for the playback identifier; A processing module is used to obtain the video frame image of the screen recording file; based on the fast forward picture experts group ffmpeg, each obtained video frame image is converted into a video in the encoding format corresponding to the version number, and the video is sent to the client so that the client can play back the video.

9. An electronic device, characterized in that: include: A memory for storing program instructions; The processor is used to call the program instructions stored in the memory, and execute the steps included in the video playback method according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the video playback method according to any one of claims 1 to 7.

Images