Pure browser preview method and system supporting multiple video formats

Abstract

The invention discloses a pure browser preview method and system supporting multiple video formats. The preview method comprises the following steps: compiling FFmpeg into a workpiece capable of running in a browser environment by using an Emscripten tool chain; when the browser does not support the format of the video file, calling a workpiece to carry out video decoding and format conversion, and outputting to the browser; wherein when the volume of the video file is greater than a preset threshold value, performing adaptive dynamic fragmentation on the video file to form a fragmentation queue, and then calling a workpiece to perform video decoding and format conversion on each fragmentation in the fragmentation queue in sequence. According to the method and the device, video playing on the browser can be realized without depending on a server, a video playing function in various formats is supported, meanwhile, the large video file can be adaptively and dynamically adjusted according to the current resource occupancy rate, and the conversion efficiency of the video is improved.

Description

Technical Field

[0001] The invention belongs to the field of browser video playback, and in particular relates to a pure browser preview method and system supporting multiple video formats. Background Art

[0002] In the current browser video playback field, although there are multiple libraries (such as Video.js, HLS.js and Dash.js) that provide video playback support, there are still the following defects: (1) Format support limitations. Most existing solutions rely on HTML5. <video>(1) The browser does not natively support certain video formats (such as AVI, MKV, etc.); (2) Compatibility issues: Different browsers have different levels of support for video formats, which makes it challenging for developers to achieve multi-platform compatibility; (3) Server dependence: Most current technologies rely on the server for video preview and transcoding. This approach not only increases the server burden, but also causes latency and bandwidth consumption problems, making the user experience of playing videos locally less than ideal.

[0003] Even though it is possible to directly play videos in various video formats on the browser side, large video files often experience playback freezes, and when the user's device capabilities are poor, there is often a problem of video playback delays, affecting the user's viewing experience. Summary of the invention

[0004] The purpose of the present invention is to provide a browser-only preview method and system supporting multiple video formats, so as to solve the problem of unsmooth video playback mentioned in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] On one hand, the present invention provides a pure browser preview method supporting multiple video formats, the preview method comprising:

[0007] Use the Emscripten toolchain to compile FFmpeg into an artifact that can run in a browser environment;

[0008] When the browser does not support the format of the video file, call the artifact to decode and convert the video format and output it to the browser;

[0009] When the size of a video file is larger than a preset threshold, the video file is adaptively and dynamically segmented to form a segment queue, and then the workpiece is called to sequentially decode and convert the video format of each segment in the segment queue;

[0010] In the adaptive dynamic segmentation, the current resource occupancy is used to calculate the current segment size, and the current segment of the video file is obtained based on the current segment size.

[0011] Preferably, the adaptive dynamic slicing includes:

[0012] Determine the theoretical size of each segment based on the preset number of basic segments and the volume of the input video file;

[0013] Adjust the current shard size based on the current resource usage of the device and the theoretical size of the current shard.

[0014] Preferably, the adjusting the current slice size according to the current resource occupancy of the device and the theoretical size of the current slice includes:

[0015] Calculate the volume adjustment factor of the current shard based on the current resource usage of the device for:

[0016] In the formula, Indicates the current memory usage. Indicates the memory usage threshold, Indicates the current CPU usage. Indicates the CPU usage threshold. Indicates the network bandwidth. Indicates the network bandwidth threshold;

[0017] Adjust the size of the current shard using the volume adjustment factor of the current shard and the theoretical size of the current shard for:

[0018]

[0019] In the formula, The theoretical size of the current shard.

[0020] Preferably, the preview method further comprises: while adaptively and dynamically segmenting the video file exceeding a preset threshold, detecting key scenes on the video file, and jumping to a corresponding position for priority segmentation when a key scene is detected.

[0021] Preferably, the key scene is characterized by a scene change score, and when the scene change score is greater than a preset threshold, a key scene exists at the corresponding position;

[0022] The scene change scoring method comprises:

[0023] Calculate the image change score based on the difference in image features between two adjacent frames;

[0024] The sum of the image change scores between two adjacent frames in the preset time window is calculated as the scene change score.

[0025] Preferably, the calculation formula for the image change score is:

[0026]

[0027] In the formula, represents the image index, , Indicates the total number of images in the preset time window. Indicates color difference, represents the motion vector, Indicates texture differences, represents the brightness difference between frames, represents the difference in edge features, They all represent the corresponding weights, represents the color richness difference threshold, represents the motion detection threshold, represents the texture difference threshold, represents the brightness difference threshold, represents the edge feature threshold, represents the adjustment factor, , Indicates the size of the video file. Indicates the preset maximum video file size.

[0028] Preferably, in the scene change scoring method, after calculating the image change score, the image change score correction step is further included:

[0029] Obtaining the scene type of the inter-frame image through a scene type discrimination strategy;

[0030] The scene type discrimination result is used to correct the corresponding image change score.

[0031] Preferably, the scene type discrimination strategy is: judging whether the scene type score is greater than 1, if so, the current scene is considered to be a dynamic scene, otherwise it is a static scene;

[0032] The calculation formula for the scene type score is:

[0033] in, represents the motion vector, represents the difference in edge features, represents the motion detection threshold, Indicates the edge feature threshold;

[0034] The correction formula of the image change score is:

[0035]

[0036] in, Score image changes, Represents the adjustment factor of the dynamic scene, Indicates the highest threshold for dynamic scene determination. represents the adjustment factor for static scenes, Indicates the minimum threshold for static scene judgment.

[0037] Another aspect of the invention provides a browser-only preview system supporting multiple video formats, the system comprising:

[0038] FFmpeg decoding artifact, which can run independently on the browser for video decoding and format conversion;

[0039] The format determination module is used to obtain the formats supported by the browser and perform format support detection on the input video file;

[0040] A fragmentation processing module, used for adaptive dynamic fragmentation processing of video files larger than a preset threshold;

[0041] The video upload module is used to obtain the video files input by the user.

[0042] Preferably, the system further comprises a key scene detection module, which is used to calculate scene change scores of consecutive frames of the video file and perform key scene detection based on the scene change scores.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) By directly porting the video decoder to the browser, it supports playback and decoding of multiple formats without relying on the server for transcoding. All you need to do is load the corresponding decoding library to play various videos smoothly in the browser;

[0045] 2) Through adaptive slicing, the size of the current slicing is automatically adjusted according to the current resource occupancy, and the most reasonable slicing size is allocated for the current resources for format conversion, which reduces resource waste, optimizes video processing efficiency, and improves viewing experience;

[0046] 3) Through key scene detection and the priority segmentation mechanism triggered under key scenes, users are provided with the function of prioritizing the viewing of key scene clips. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the browser-only preview method of Embodiment 1.

[0048] Figure 2 This is a flowchart of the browser-only preview method of Embodiment 2.

[0049] Figure 3 This is a flow chart of the adaptive dynamic slicing in the second embodiment.

[0050] Figure 4 The present invention is a schematic diagram of the framework structure of a pure browser preview system supporting multiple video formats. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0052] Reference Figure 1 As shown, a pure browser preview method supporting multiple video formats in this embodiment includes browser pre-operation and video processing flow.

[0053] The browser's pre-processing operation is to compile the FFmpeg decoder into an efficient artifact in bytecode format using the Emscripten tool chain, enabling it to run independently in the browser environment and improving the browser's video processing speed.

[0054] Emscripten is a compiler based on LLVM. In theory, it can compile any code that can generate LLVM bitcode into asm.js, a strict subset of javascript. In practice, it is mainly used to compile C / C++ code into asm.js. FFmpeg is a cross-platform open source software that can be used for video encoding, decoding and format conversion. The compilation process ensures that FFmpeg can achieve cross-platform compatibility and optimize video decoding and format conversion performance.

[0055] The present invention supports the decoding and playback of multiple video formats, including MP4, AVI, MKV, etc., by transplanting the FFmpeg decoder to the browser side. At the same time, the video decoding and playback are completely completed on the client side, getting rid of the dependence on the server. Even in the case of limited bandwidth or unstable network, it can still maintain smooth playback performance. This artifact can be used with the browser's own player and HTML5 <video>The seamless integration of tags enables developers to operate through familiar APIs, reducing learning costs and eliminating the need to relearn or adjust existing development methods.

[0056] The video processing flow includes:

[0057] Step 1: Input of user video files.

[0058] Step 2, detect the browser's compatibility with different video processing technologies, obtain the efficient execution technology supported by the browser, and use the efficient execution technology for subsequent operations, otherwise fall back to other compatible processing methods, such as browser non-support prompts or browser change prompts.

[0059] Here, efficient execution technology refers to API calls such as WebAssembly, WebCodecs, WEBGPU, etc. Different browsers support different underlying efficient execution technologies. For example, Firefox supports WebAssembly, WebGPU, and partially supports WebCodes, while Safari supports WebAssembly but not WebCodes.

[0060] Step 3, performing format support detection on the video file. If the browser can support the native format or encoding of the video file, the browser directly plays the video file, otherwise, execute step 4.

[0061] Step 4, determine whether the size of the video file is greater than a preset threshold. If so, perform adaptive dynamic segmentation on the video file to form a segment queue, call the artifact to use the efficient execution technology supported by the browser to perform multi-threaded video decoding and format conversion on each segment in the segment queue in sequence, otherwise directly call the artifact to use the efficient execution technology supported by the browser to perform video decoding and format conversion on the video file.

[0062] In step 4 of the present invention, the size of the video file is judged by its volume. If it is a small file, video decoding and format conversion are directly performed. If it is a large file, it needs to be fragmented first. During the fragmentation process, the current fragment size of the video is adjusted according to the current performance of the device and the browser, so that the resource utilization at each moment is optimal, so that the format conversion speed of the video clip under the optimal resource utilization is also optimal, so as to improve the video conversion performance. At the same time, the fragments are converted simultaneously by multiple threads to ensure that the format or encoding is converted in a very short time, thereby reducing the user's waiting time and improving the loading speed.

[0063] In this adaptive dynamic sharding, the threshold of the number of shards is first set, and then the size of the current shard is dynamically calculated based on the threshold and the resource occupancy at that time. While ensuring that the total number of shards is close to the threshold of the number of shards, the size of each shard can be dynamically adjusted according to the resource occupancy to optimize the shard processing efficiency. The specific process is as follows:

[0064] Step 4.1, based on the preset number of basic shards and the size of the input video file Determine the theoretical size of each shard as:

[0065] Step 4.2, calculate the volume adjustment coefficient of the current shard based on the current resource usage:

[0066]

[0067] In the formula, Indicates the current memory usage. Indicates the memory usage threshold, Indicates the remaining memory ratio. Indicates the current CPU usage. Indicates the CPU usage threshold. Indicates the remaining CPU ratio. Indicates the network bandwidth. Indicates the network bandwidth threshold. Indicates the network bandwidth ratio. ;

[0068] Step 4.3, adjust the size of the current shard using the volume adjustment coefficient of the current shard and the theoretical size of the current shard for:

[0069] .

[0070] Since each shard requires a separate file handle, too many shards will cause a large amount of occupancy, and the processing of each shard requires the allocation of a certain amount of memory. Too many shards will occupy memory resources and aggravate memory fragmentation. The theoretical size of each shard calculated by step 4.1 of the present invention is the same, and dynamic fine-tuning is performed on different shards based on the same volume to ensure the reliability of the shards and prevent the number of shards far exceeding the threshold when the user's device capabilities are poor. In this adaptive dynamic algorithm, when resources are sufficient, Larger, the shard size is closer to the base value; when resources are tight, Smaller, the shard size will be reduced.

[0071] In step 4.2 of the present invention, by taking memory, network bandwidth, and CPU occupancy rate as calculation parameters of resource occupancy, rather than only taking single parameters such as memory and CPU occupancy rate as the representation of resource occupancy rate, potential performance bottlenecks can be better identified and the accuracy of resource occupancy representation can be ensured; by taking memory, network bandwidth, and CPU occupancy rate together to calculate the adjustment coefficient of the shard volume, when a certain resource is overloaded, measures such as reducing the number of shards, adjusting the shard size, etc. can be taken to reduce the pressure on bottleneck resources, avoid the risk of performance degradation or system crash, and improve the scalability of the system. When the external load changes, the system can dynamically adjust the shards according to the resource usage to ensure that resources are reasonably allocated and used.

[0072] Step 5: Generate a Blob URL to return the converted streaming media data in real time for users to play instantly on the browser.

[0073] Blob URL is a special URL generated by the browser for temporary access to binary data (such as pictures, videos, etc.) stored in the memory. After the blob URL is directly assigned to the video tag or player module, the browser can play the video. The method of generating the Blob URL is common knowledge in the art and will not be elaborated here.

[0074] In step 5 of the present invention, during the process of converting the video file into segments, the user can achieve real-time streaming playback of the video through the generated Blob URL. This design allows the user to watch the video while converting the video, greatly improving the response speed of the playback.

[0075] Step 6: merge all the converted segments to generate a new video file for users to download or play directly in the browser.

[0076] In the present invention, whether the browser can support the native format or encoding of the video file, or after the video file is decoded and format converted through the artifact, the browser will use the efficient execution technology detected in step 2 to play the video. The modularly designed artifact can use different underlying technologies for different browsers to accelerate, which can rely on the native functions of the browser or use efficient compilation modules to improve the overall processing capability.

[0077] Reference Figure 2 As shown, the above-mentioned pure browser preview method is improved: in step 4, while adaptively and dynamically segmenting the video files exceeding the preset threshold, key scene detection is performed on the video files, and when the key scene is detected, the video file is jumped to the corresponding position for priority segmentation.

[0078] In the present invention, when a key scene is detected, it is necessary to determine whether the detected key scene has been fragmented. If it has been fragmented, no secondary fragmentation processing is performed. Otherwise, jump to the corresponding position of the key scene for priority fragmentation. After the priority fragmentation processing of the key scene is completed, it is still necessary to return to the previous fragmentation task and wait for the next jump to be performed after a new key scene appears.

[0079] In this embodiment, the key scene is characterized by a scene change score. When the scene change score is greater than a preset threshold, it indicates that the video clip corresponding to the scene change score is a key scene.

[0080] In step 4 of the present invention, each time a fragment is calculated and processed, the fragment queue is updated. Figure 3 As shown, for example, when thread A adaptively and dynamically segments the video file in chronological order to obtain segments 1 to 11, thread B detects that the scene change score of the video file from 12:05 to 12:30 is greater than the preset threshold, and 12:05 is marked at this time. Thread A determines the starting point of the current segment at the 20MB position of the video based on the marked point, and uses it as the starting point of the current segment. After calculating the size of the current segment through steps 4.1-4.3, segmentation is performed to obtain segment 12, and then adaptive dynamic segmentation is performed in chronological order. It should be noted that in the subsequent adaptive dynamic segmentation, the video segment corresponding to segment 12 is no longer segmented.

[0081] In the present invention, by setting a priority segmentation system for key scenes, it is ensured that users can play key scene clips immediately. Through the instant playback function of the key video clips, another new viewing option is provided for users, improving the user experience.

[0082] As a specific implementation method of the scene change scoring, the specific process is as follows:

[0083] (a) Calculate the image change score based on the difference in image features between two adjacent frames , Represents the image frame index;

[0084] (b) Calculate the sum of the image change scores between two adjacent frames in the preset time window as the scene change score, denoted as .

[0085] Here, the similarity between two adjacent image frames is calculated by the difference in image features, and then the sum of the similarities of consecutive frames is used to characterize the scene change score of the video clip. The higher the scene change score, the more information is covered in the corresponding video clip, and the video clip is considered to meet the user's needs. The preset time window is used to fix the number of image change scores when calculating the scene change score. The time window is a sliding time window, and the moving step size can be set according to the actual situation.

[0086] Here, the image change score is calculated by the following formula:

[0087]

[0088] In the formula, represents the image index, , Indicates the total number of images in the preset time window. Indicates color difference, represents the motion vector, Indicates texture differences, represents the brightness difference between frames, represents the difference in edge features, They all represent the corresponding weights, represents the color richness difference threshold, represents the motion detection threshold, represents the texture difference threshold, represents the brightness difference threshold, represents the edge feature threshold, represents the adjustment factor, , Indicates the video file size. Indicates the preset maximum video file size; the method for obtaining the above parameters is a conventional technical means in this field and will not be described here.

[0089] In this embodiment, the scene change score is calculated by combining visual features such as color, motion, texture, brightness, and edges to ensure the reliability of the scene change score, and a weight coefficient that can be dynamically adjusted according to user needs is introduced to meet different user needs and improve the user's viewing experience.

[0090] In this embodiment, by setting an adjustment factor, scoring deviations caused by different video file sizes are avoided, helping the algorithm to better adapt to video inputs of different qualities and sizes, ensuring accurate detection of scene changes under different conditions, and avoiding misjudgments and performance degradation due to excessive sensitivity when processing large files. This is more important for complex and changeable video content and helps improve the overall performance of scene change detection.

[0091] Furthermore, the scene change score in this embodiment is optimized: after obtaining the image change score, the dynamic scene and the static scene are judged by the scene type discrimination strategy to obtain the scene judgment result, and the image change score is corrected using the score correction strategy under different scenes, and the corrected image change score is used for calculation in the scene change score.

[0092] In this scene type discrimination strategy, if the scene type score is greater than 1, it is considered that the scene with dynamic changes between the current frame image and the next frame image is a dynamic scene, otherwise it is a static scene; the calculation of the scene type score is shown in the following formula:

[0093]

[0094] In the formula, represents the motion vector, represents the difference in edge features, represents the motion detection threshold, Represents the edge feature threshold.

[0095] Here, by normalizing the changes in motion and edge features, the degree of change in different video contents and environments can be fairly compared to determine the scene type.

[0096] The image change score correction strategy under different scenarios is as follows:

[0097]

[0098] in, Score image changes, Represents the adjustment factor of the dynamic scene, Indicates the highest threshold for dynamic scene determination. represents the adjustment factor for static scenes, Indicates the minimum threshold for static scene judgment.

[0099] Here, the original score is adjusted by scene type to more accurately reflect the degree of scene change; in dynamic scenes, the score is amplified to detect changes more sensitively; in static scenes, the score is reduced to avoid misjudging minor changes as scene changes.

[0100] Reference Figure 4 As shown, the present invention also provides a pure browser preview system supporting multiple video formats, including:

[0101] Video upload module, used to obtain video files input by users;

[0102] The format determination module is used to obtain the video formats supported by the browser and perform format support detection on the video files input by the user obtained by the video upload module;

[0103] A fragmentation processing module, used for adaptively and dynamically fragmenting the video file when the volume of the video file input by the user is larger than a preset threshold and the browser does not support the format of the video file input by the user;

[0104] A key scene detection module, used to calculate scene change scores of consecutive frames of a video file, and perform key scene detection based on the scene change scores;

[0105] The trigger mechanism triggers the fragment processing module to jump to the corresponding position for priority fragment processing when a key scene is detected;

[0106] FFmpeg decoding artifact, which can run independently on the browser and is used to decode and convert video files or fragments.

[0107] In the format judgment module, when the browser supports the format of the video file input by the user, the browser directly plays the video. The format judgment module is connected to the FFmpeg decoding artifact and the fragmentation processing module respectively. At the same time, the fragmentation processing module is connected to the FFmpeg decoding artifact. When the video file is too large, it is first fragmented and then the format is converted. When the video file is not large, the format is directly converted and the browser directly plays the video.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / video> < / video>

Claims

1. A browser-only preview method supporting multiple video formats, characterized in that: The preview method includes Use the Emscripten toolchain to compile FFmpeg into an artifact that can run in a browser environment; When the browser does not support the format of the video file, call the artifact to decode and convert the video format and output it to the browser; When the size of a video file is larger than a preset threshold, the video file is adaptively and dynamically segmented to form a segment queue, and then the workpiece is called to sequentially decode and convert the video format of each segment in the segment queue; In the adaptive dynamic segmentation, the current resource occupancy is used to calculate the current segment size, and the current segment of the video file is obtained based on the current segment size.

2. A browser-only preview method supporting multiple video formats as claimed in claim 1, characterized in that: The adaptive dynamic slicing comprises: Determine the theoretical size of each segment based on the preset number of basic segments and the volume of the input video file; Adjust the current shard size based on the current resource usage of the device and the theoretical size of the current shard.

3. A browser-only preview method supporting multiple video formats as claimed in claim 2, characterized in that: The adjusting of the current slice size according to the current resource occupancy of the device and the theoretical size of the current slice includes: Calculate the volume adjustment factor of the current shard based on the current resource usage of the device for: In the formula, Indicates the current memory usage. Indicates the memory usage threshold, Indicates the current CPU usage. Indicates the CPU usage threshold. Indicates the network bandwidth. Indicates the network bandwidth threshold; Adjust the size of the current shard using the volume adjustment factor of the current shard and the theoretical size of the current shard for: In the formula, The theoretical size of the current shard.

4. A browser-only preview method supporting multiple video formats as claimed in claim 3, characterized in that: The preview method further includes: while adaptively and dynamically slicing the video file exceeding a preset threshold, detecting key scenes on the video file, and jumping to a corresponding position for priority slicing when a key scene is detected.

5. A browser-only preview method supporting multiple video formats as claimed in claim 4, characterized in that: The key scene is characterized by a scene change score. When the scene change score is greater than a preset threshold, a key scene exists at the corresponding position; The scene change scoring method comprises: Calculate the image change score based on the difference in image features between two adjacent frames; The sum of the image change scores between two adjacent frames in the preset time window is calculated as the scene change score.

6. A browser-only preview method supporting multiple video formats as claimed in claim 5, characterized in that: The calculation formula of the image change score is: In the formula, represents the image index, , Indicates the total number of images in the preset time window. Indicates color difference, represents the motion vector, Indicates texture differences, represents the brightness difference between frames, represents the difference in edge features, They all represent the corresponding weights, represents the color richness difference threshold, represents the motion detection threshold, represents the texture difference threshold, represents the brightness difference threshold, represents the edge feature threshold, represents the adjustment factor, , Indicates the size of the video file. Indicates the preset maximum video file size.

7. A browser-only preview method supporting multiple video formats as claimed in claim 5, characterized in that: In the scene change scoring method, after the image change score is calculated, a correction step of the image change score is also included: Obtaining the scene type of the inter-frame image through a scene type discrimination strategy; The scene type discrimination result is used to correct the corresponding image change score.

8. A browser-only preview method supporting multiple video formats as claimed in claim 7, characterized in that: The scene type discrimination strategy is: determine whether the scene type score is greater than 1, if so, the current scene is considered to be a dynamic scene, otherwise it is a static scene; The calculation formula for the scene type score is: in, represents the motion vector, represents the difference in edge features, represents the motion detection threshold, Indicates the edge feature threshold; The correction formula of the image change score is: in, Score image changes, Represents the adjustment factor of the dynamic scene, Indicates the highest threshold for dynamic scene determination. represents the adjustment factor for static scenes, Indicates the minimum threshold for static scene judgment.

9. A browser-only preview system supporting multiple video formats, characterized in that: The system comprises: FFmpeg decoding artifact, which can run independently on the browser for video decoding and format conversion; The format determination module is used to obtain the formats supported by the browser and perform format support detection on the input video file; A fragmentation processing module, used for adaptive dynamic fragmentation processing of video files larger than a preset threshold; The video upload module is used to obtain the video files input by the user.

10. A browser-only preview system supporting multiple video formats as claimed in claim 9, characterized in that: The system also includes a key scene detection module, which is used to calculate scene change scores of continuous frames in the video file and perform key scene detection based on the scene change scores.

Images