Video processing method and device and computer storage medium

Abstract

The invention provides a video processing method and device and a computer storage medium, the video processing method is applied to a client, and the video processing method comprises the following steps: obtaining historical operation parameters; acquiring playing information of the current recorded video; determining a pre-reading parameter based on the historical operation parameter and the playing information; and uploading the pre-reading parameter to a video cloud to obtain a video stream reconstructed by the video cloud according to the pre-reading parameter. Through the above mode, through the self-adaptive adjustment pre-reading parameter generation mode, the video on demand playback fast response is improved, and the video downloading speed is accelerated.

Description

Technical Field

[0001] The present application relates to the technical field of video processing, and particularly to a video processing method, apparatus, and computer storage medium. Background Art

[0002] In the prior art, a video playback system usually downloads data sequentially. Conventional prefetching methods cannot adapt to different operation habits, and in scenarios where it is necessary to frequently switch the progress bar to view a video or quickly download a video, problems such as low buffer rate and high latency will occur. Summary of the Invention

[0003] The present application provides a video processing method, apparatus, and computer storage medium.

[0004] To solve the above technical problems, the present application proposes a video processing method. The video processing method is applied to a client, and the video processing method includes: obtaining historical operation parameters; obtaining playback information of a current recorded video; determining prefetch parameters based on the historical operation parameters and the playback information; and uploading the prefetch parameters to a video cloud to obtain a video stream reconstructed by the video cloud according to the prefetch parameters.

[0005] Among them, the obtaining of the historical operation parameters includes: obtaining model parameters from the video cloud based on preset information; initializing a behavior detection module with the model parameters; and inputting a current operation into the behavior detection module to obtain the historical operation parameters.

[0006] Among them, the prefetch parameters include: on-demand position, data prefetch length, number of prefetch points, and / or prefetch point distribution position coefficient.

[0007] Among them, the determining of the prefetch parameters based on the historical operation parameters and the playback information includes: determining a progress bar time range, bitstream size, and on-demand position of the current recorded video based on the playback information; determining the number of prefetch points according to the progress bar time range; determining the data prefetch length according to the bitstream size; and determining the prefetch point distribution position coefficient based on the historical operation parameters.

[0008] Among them, after determining the prefetch point distribution position coefficient based on the historical operation parameters, the video processing method includes: determining the number of prefetch data blocks on the left side and the number of prefetch data blocks on the right side of the on-demand position based on the prefetch point distribution position coefficient and the number of prefetch points.

[0009] To solve the above technical problems, the present application proposes a video processing method, which is applied to a video processing system. The video processing system includes a client and a video cloud. The video cloud includes a streaming media module and a storage system. The video processing method includes: the client obtains historical operation parameters; the client obtains the playback information of the current recorded video; the client determines prefetch parameters based on the historical operation parameters and the playback information and uploads them to the video cloud; the streaming media module reads a plurality of data shards from the storage system according to the prefetch parameters; the streaming media module sorts and concatenates the plurality of data shards according to timestamps to obtain a video stream and sends it to the client.

[0010] Among them, the streaming media module reads a plurality of data shards from the storage system according to the prefetch parameters, including: the streaming media module constructs a relative number of threads according to the number of prefetch points in the prefetch parameters; the streaming media module reads the data shards from the storage system through each thread.

[0011] Among them, the streaming media module reads the data shards from the storage system through each thread, including: the streaming media module determines the prefetch point offset of each data shard according to the data prefetch length and the number of prefetch points in the prefetch parameters; the streaming media module obtains the current offset position of the current on-demand position in the file of the current recorded video; the streaming media module determines the prefetch offset position of each data shard in the file according to the current offset position and the prefetch point offset of each data shard; the streaming media module reads the file data at the prefetch offset position from the storage system through each thread to read the data shards.

[0012] To solve the above technical problems, the present application proposes a video processing device. The video processing device includes a memory and a processor coupled to the memory. Among them, the memory is used to store program data, and the processor is used to execute the program data to implement the above video processing method.

[0013] To solve the above technical problems, the present application proposes a computer storage medium. The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the above video processing method.

[0014] Distinct from the prior art, the beneficial effects of the present application are as follows: The video processing device obtains historical operation parameters; obtains the playback information of the current recorded video; determines prefetch parameters based on the historical operation parameters and the playback information; and uploads the prefetch parameters to the video cloud to obtain the video stream reconstructed by the video cloud according to the prefetch parameters. In the above manner, through an adaptive adjustment method for generating prefetch parameters, the fast response of video-on-demand playback is improved and the video download speed is accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0016] Figure 1 is a schematic flowchart of the first embodiment of the video processing method provided by the present application;

[0017] Figure 2 is a schematic diagram of conventional prefetching provided by the present application;

[0018] Figure 3 is a schematic diagram of random-on-demand type prefetching, multi-threaded reading, and data merging provided by the present application;

[0019] Figure 4 is a schematic diagram of sequential-play type prefetching, multi-threaded reading, and data merging provided by the present application;

[0020] Figure 5 is a schematic flowchart of the second embodiment of the video processing method provided by the present application;

[0021] Figure 6 is a schematic diagram of the video processing system provided by the present application;

[0022] Figure 7 is a schematic flowchart of the third embodiment of the video processing method provided by the present application;

[0023] Figure 8 is a schematic structural diagram of an embodiment of the video processing device provided by the present application;

[0024] Figure 9 is a schematic structural diagram of an embodiment of the computer storage medium provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0026] Among them, the video processing method of the present application is applied to a video processing device. The video processing device of the present application can be a server or a system in which the server and the local terminal cooperate with each other. Correspondingly, each part included in the video processing device, such as each unit, sub-unit, module, and sub-module, can be all set in the server or can be respectively set in the server and the local terminal.

[0027] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or can be implemented as a single software or software module, which is not specifically limited herein. In some possible implementation manners, the video processing method of the embodiments of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0028] The present application proposes a video processing method. In this embodiment, the video processing method is applied to a client. Please refer to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the video processing method provided by the present application.

[0029] As Figure 1 shown, the specific steps are as follows:

[0030] Step S11: Obtain historical operation parameters.

[0031] Among them, the historical operation parameters are operation habit parameters generated by operating the video several times.

[0032] The historical operation parameters include but are not limited to the playback type, the video recording query time range, and the magnitude of the progress bar drag.

[0033] Among them, the playback type includes the random on-demand type A or the sequential playback type B. The two types are represented by percentages, and the sum is 1 or 100%.

[0034] Among them, the video recording query time range is the query time span duration in the file after each on-demand trigger.

[0035] Among them, the dragging amplitude of the progress bar, that is, the most likely span size when dragging the progress bar once, is measured by the length of time on the client side and converted to be measured by data offset and length in the file.

[0036] In a specific embodiment of the present application, the video processing device obtains model parameters from the video cloud based on preset information; initializes the behavior detection module with the model parameters; inputs the current operation into the behavior detection module to obtain the historical operation parameters.

[0037] Among them, the preset information is the user information stored on the client.

[0038] Specifically, when the client plays the video, the client actively calls the cloud storage SDK interface through the streaming media to obtain the current persisted historical training results, including the model parameters obtained from training and some metadata, such as the learning rate, model structure, optimizer state, etc. The model parameters are stored in the standard file format in the SavedModel format of TensorFlow (an open-source machine learning framework). When loading, the same TensorFlow library is used to read the saved model file, reconstruct the model structure, and restore the training state.

[0039] The behavior detection module detects the operation data of the current client interface, including the current operation time, click to play, drag the progress bar, zoom the progress bar, channel information, etc., to form the client operation process and realize continuous training of the model.

[0040] Step S12: Obtain the playing information of the current video recording.

[0041] Among them, the playing information of the video includes but is not limited to video channel information, video file information, the playing position of the current video recording, the position of the mouse in the progress bar, etc.

[0042] Among them, the position of the mouse in the progress bar is obtained at the starting offset position of reading data in the file.

[0043] Step S13: Determine the prefetch parameters based on the historical operation parameters and the playing information.

[0044] Among them, the prefetch parameters include: on-demand position, data prefetch length, number of prefetch points, and / or prefetch point distribution position coefficient.

[0045] Among them, the on-demand position is the on-demand position of the client video recording progress bar, which is reflected as the offset in the file in the cloud storage SDK. After opening the file for the first time, seek to this position to start reading data.

[0046] In an embodiment of the present application, the video processing device determines the progress bar time range, bitstream size, and on-demand position of the current recorded video based on the playback information; determines the number of pre-read points according to the progress bar time range; determines the data pre-read length according to the bitstream size; and determines the pre-read point distribution position coefficient based on the historical operation parameters.

[0047] Among them, the data pre-read length is the data pre-read length read at the on-demand position, and the length of the first pre-read is adjusted according to the size of the current bitstream. If the bitrate is 2 Mb / s, the average key frame is less than 50 KB. If the bitrate is 4 Mb / s, the average key frame is less than 100 KB. If the bitrate is 8 Mb / s, the average key frame is less than 700 KB. The first pre-read data is controlled to be 4 times the average key frame size according to the key frame sizes corresponding to the above different bitrates, ensuring that the pre-read data includes key frames and enabling the client to quickly display images for the first time.

[0048] Among them, the current number of pre-read points N is determined by the progress bar time range, with a default maximum of 8. It directly determines the number of dynamic read threads. To avoid too fine a granularity, the time span of each pre-read data block is at least 30 s and at most 120 s.

[0049] Among them, the pre-read point distribution position coefficient X is determined by the biased playback type. The maximum value of random pre-read is X = 0.5. If it exceeds 0.5, it is considered a random pre-read scenario. For example, for the random on-demand type 0.5 and sequential playback 0.5, the number of pre-reads on the left side of the on-demand position is 8 * 0.5 = 4, and the same for the right side, evenly distributed on both sides of the on-demand position. If the random on-demand type is 0.2, the number of pre-read data blocks on the left side of the on-demand is 8 * 0.2 = 1.6, and after rounding, it is 1. So, the maximum number of pre-reads on the right side of the on-demand position is 7.

[0050] Among them, the pre-read time length of each pre-read point is determined by the time span T0 of the current client progress bar. The duration of each time slice is T0 / the number of pre-read points N, and the maximum value of N is 8. If it does not meet the minimum of 30 s, the total number of pre-read points is reduced, and finally, after being passed to the streaming media, it is converted into the data length in the file according to the file information.

[0051] As Figures 2 - 4 shown, Figure 2 is the schematic diagram of conventional pre-read provided by the present application; Figure 3 is the schematic diagram of random on-demand type pre-read, multi-threaded reading, and data merging provided by the present application; Figure 4 is the schematic diagram of sequential playback type pre-read, multi-threaded reading, and data merging provided by the present application.

[0052] In the prior art, the conventional prefetching process starts from the position read this time, reads and caches more data than the target requirement. When reading next time, if the cache is hit, the reading speed can be improved. However, if there is a drag progress bar instruction, data not in the cache will be frequently read, resulting in a slow speed. Or when reading high-bitrate data, the computing efficiency of a single thread is low.

[0053] Random on-demand is biased towards random on-demand instructions, and the prefetch points are generally evenly distributed before and after the on-demand position; sequential on-demand is biased towards sequential play instructions, and the prefetch points are generally distributed after the on-demand position.

[0054] This application proposes an adaptive prefetching algorithm. The video processing device detects the play type corresponding to the play instruction. If it is biased towards the random on-demand type A, it is necessary to ensure that data clicked at any position of the progress bar on the current client screen can be quickly read and played. If it is biased towards the sequential play type B, it is necessary to ensure that data after the current time point can be quickly read and played as much as possible.

[0055] This application is different from the prior art. The current mouse on-demand position is preferentially read. Different from other prefetch point prefetching strategies, the data reading speed at the on-demand position in this application directly affects the client output response speed.

[0056] This application determines the usage habit parameters according to the historical operations provided by the model, and combines the current channel information, that is, the video file information, the position of the mouse on the progress bar, and the starting offset position for reading data in the file, to determine the initial offset of the file to be read this time in the cloud storage, and calculates the prefetch parameters. The prefetch starting positions are 1 - 8, the number of prefetch data blocks is 1 - 8, and the prefetch length of each data block. The prefetch parameters control which positions the next SDK should start prefetching which range of data from the cloud storage.

[0057] In the embodiment of this application, after determining the prefetch point distribution position coefficient based on the historical operation parameters, the video processing method includes: determining the number of prefetch data blocks on the left side and the number of prefetch data blocks on the right side of the on-demand position based on the prefetch point distribution position coefficient and the number of prefetch points.

[0058] Step S14: Upload the prefetch parameters to the video cloud to obtain the video stream reconstructed by the video cloud according to the prefetch parameters.

[0059] Specifically, the client uploads the current 1 - 8 on-demand positions and the prefetch parameters to the video cloud, and the video stream reconstructed according to the prefetch parameters is sent to the streaming media after being sent back.

[0060] The streaming media downloads data blocks, i.e., segmented video clips, through the video cloud storage SDK. First, it finds the index information of the video file from the cloud storage metadata, opens the logical file, and then starts from the corresponding file offset for on-demand playback. According to the issued prefetch configuration, it adjusts the prefetch parameters in the memory, asynchronously reads the fixed-length data packet header and prefetch data. For the prefetch length after the on-demand position, the client adjusts the prefetch length according to the bitrate size to ensure that the first prefetch content includes the first key frame data after the current position, thereby ensuring fast output for the first on-demand playback.

[0061] In the embodiments of the present application, the adaptive prefetch algorithm based on operation behavior adopts machine learning technology to analyze the viewing behavior habits of historical video recordings and predict future viewing needs, thereby realizing the adaptive prefetch of videos.

[0062] To solve the above technical problems, the present application proposes a video processing method, which is applied to a video processing system. The video processing system includes a client and a video cloud.

[0063] As Figures 5 - 6 shown, Figure 5 is a schematic flowchart of the second embodiment of the video processing method provided by the present application. Figure 6 is a schematic diagram of the video processing system provided by the present application.

[0064] As Figure 6 shown, in the video processing system in the embodiments of the present application, according to the video reading function, it can be generally divided into three layers. The top layer is the video playback client, which mainly provides functions such as video on-demand playback and download, and integrates a behavior detection module and a lightweight machine learning model inside. The behavior detection module mainly detects and records the structured information of the video channel, time range, progress bar unit (hours, minutes, seconds), behavior sequence (click, drag, zoom), etc. when viewing or dragging the progress bar. The middle layer is the streaming media module. Since it is a distributed cluster, there are multiple streaming media. Each streaming media interacts with the video cloud storage cluster through the cloud storage SDK to complete functions such as data upload and download. A video splicing module is integrated inside the streaming media to realize the parsing of the data blocks read from the cloud storage and the splicing according to the time stamp sequence to restore the complete video file data. The bottom layer is a distributed video cloud storage system, which mainly stores video data and machine learning model parameters and other persistent information through the erasure code method.

[0065] As Figure 5 shown, the specific steps are as follows:

[0066] Step S21: The client obtains historical operation parameters.

[0067] Step S22: The client obtains the playback information of the current video recording.

[0068] Step S23: The client determines prefetch parameters based on the historical operation parameters and the playback information, and uploads them to the video cloud.

[0069] Steps S21 - S23 are the same as Steps S11 - S14, and will not be elaborated here.

[0070] Step S24: The streaming media module reads a number of data shards from the storage system according to the prefetch parameters.

[0071] An embodiment is proposed in this application for reading data shards. For details, please refer to Figure 7 , Figure 7 which is a schematic flowchart of the third embodiment of the video processing method provided by this application.

[0072] As Figure 7 shown, the specific steps are as follows:

[0073] Step S31: The streaming media module constructs a relative number of threads according to the number of prefetch points in the prefetch parameters.

[0074] Specifically, the prefetch parameter client sends the parameters to the streaming media module through the TCP protocol. The streaming media calls the cloud storage SDK interface and passes them into the SDK.

[0075] Step S32: The streaming media module reads the data shards from the storage system through each thread.

[0076] Specifically, the streaming media module constructs a relative number of threads according to the number of prefetch points in the prefetch parameters; the streaming media module reads the data shards from the storage system through each thread.

[0077] The SDK opens the logical file in read-only mode from the cloud storage metadata, first finds the corresponding file offset position Offset0 according to the on-demand position. When reading the packet header, it configures the prefetch length of the data read after the on-demand position into the SDK prefetch process; the data read for the first time includes the first key frame, and the client on-demand position can respond quickly for output.

[0078] The SDK takes N threads from the thread pool according to the current number of prefetch points N issued, and performs the data block reading task.

[0079] Step S25: The streaming media module sorts and splices the several data shards according to the timestamps, obtains the video stream, and sends it to the client.

[0080] The video processing device pre-reads the file data length L corresponding to the pre-read time length of each pre-read point. The offset position of the adjacent pre-read point in the file is obtained by offsetting L forward (N - L) or backward (N + L) from the on-demand point offset position. Therefore, the offset of the first pre-read point is (Offset0 – n0*L). Similarly, the offsets of other pre-read points can be calculated, and the pre-read tasks are sent to the corresponding pre-read threads. Each pre-read thread opens the file in read-only mode and then starts reading data blocks from the cloud storage according to the pre-read offset and length. If a data shard read fails in a data block, that is, the number of failed data shards is greater than the maximum tolerance of the current erasure code, it will be retried up to 3 times for local data shard retries. After successful reading, the data block is parsed for header data in the streaming media module. After the streaming media reads the packet header, it finds the position of the first key frame, finds the first key frame information from the data block, sorts and reorganizes all data blocks according to the time stamps of the data stream, and the video processing device sends the reorganized video stream to the client.

[0081] To implement the video processing method of the above embodiments, the present application also provides a video processing device. For details, please refer to Figure 8 , Figure 8 which is a schematic structural diagram of another embodiment of the video processing device provided by the present application.

[0082] As Figure 8 shown, the video processing device 600 of this embodiment includes a processor 61, a memory 62, an input / output device 63, and a bus 64.

[0083] The processor 61, the memory 62, and the input / output device 63 are respectively connected to the bus 64. The memory 62 stores a computer program, and the processor 61 is configured to execute the computer program to implement the video processing method of the above embodiments.

[0084] In this embodiment, the processor 61 can also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 61 can also be a GPU (Graphics Processing Unit), also known as a display core, visual processor, display chip, which is a microprocessor specialized for image computing on computers, workstations, game consoles, and some mobile devices (such as tablets, smartphones, etc.). The purpose of the GPU is to convert and drive the display information required by the computer system and provide a line scan signal to the display to control the correct display of the display. It is an important component connecting the display and the computer motherboard. The graphics card, as an important part of the computer host, undertakes the task of outputting and displaying graphics. The general-purpose processor can be a microprocessor or the processor 61 can also be any conventional processor, etc.

[0085] This application also provides a computer storage medium, such as Figure 9 shown, the computer storage medium 700 is used to store a computer program 71. When the computer program 71 is executed by the processor, it is used to implement the method described in the video processing method embodiment of this application.

[0086] The method involved in the video processing method embodiment of this application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0087] The above are only the embodiments of the present invention, and do not thus limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall similarly be included within the patent protection scope of the present invention.

Claims

1. A video processing method, characterized in that: The video processing method is applied to a client, and the video processing method includes: Get historical operation parameters; Get the playback information of the current recorded video; Determining pre-reading parameters based on the historical operation parameters and the playback information; The pre-read parameters are uploaded to the video cloud to obtain a video stream reconstructed by the video cloud according to the pre-read parameters.

2. The video processing method according to claim 1, characterized in that: The obtaining of historical operation parameters includes: Acquiring model parameters from the video cloud based on preset information; Initializing a behavior detection module using the model parameters; The current operation is input into the behavior detection module to obtain the historical operation parameters.

3. The video processing method according to claim 1, characterized in that: in, The pre-reading parameters include: on-demand position, data pre-reading length, number of pre-reading points, and / or pre-reading point distribution position coefficient.

4. The video processing method according to claim 3, characterized in that: The determining of the pre-reading parameters based on the historical operation parameters and the playback information includes: Based on the playback information, determine the progress bar time range, bitstream size and on-demand position of the current recorded video; Determining the number of pre-reading points according to the time range of the progress bar; Determining the data pre-read length according to the code stream size; Based on the historical operation parameters, the pre-reading point distribution position coefficients are determined.

5. The video processing method according to claim 4, characterized in that: After determining the pre-reading point distribution position coefficient based on the historical operation parameters, the video processing method includes: Based on the pre-reading point distribution position coefficient and the pre-reading point quantity, the left pre-reading data block quantity and the right pre-reading data block quantity of the on-demand position are determined.

6. A video processing method, characterized in that: The video processing method is applied to a video processing system, wherein the video processing system includes a client and a video cloud, wherein the video cloud includes a streaming media module and a storage system; The video processing method comprises: The client obtains historical operation parameters; The client obtains the playback information of the current recorded video; The client determines the pre-reading parameters based on the historical operation parameters and the playback information and uploads the pre-reading parameters to the video cloud; The streaming media module reads a number of data slices from the storage system according to the pre-read parameters; The streaming media module sorts and splices the data segments according to timestamps, obtains the video stream and sends it to the client.

7. The video processing method according to claim 6, characterized in that: The streaming media module reads a plurality of data slices from the storage system according to the pre-read parameters, including: The streaming media module constructs a relative number of threads according to the number of pre-reading points in the pre-reading parameters; The streaming media module reads the data slices from the storage system through each thread.

8. The video processing method according to claim 7, characterized in that: The streaming media module reads the data slices from the storage system through each thread, including: The streaming media module determines the pre-reading point offset of each data segment according to the data pre-reading length and the number of pre-reading points in the pre-reading parameters; The streaming media module obtains the current offset position of the current on-demand position in the file of the current recorded video; The streaming media module determines the pre-reading offset position of each data segment in the file according to the current offset position and the pre-reading point offset of each data segment; The streaming media module reads the file data at the pre-read offset position from the storage system through each thread to read the data segment.

9. A video processing device, characterized in that: The video processing device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the video processing method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the video processing method according to any one of claims 1 to 8.

Images