A Pure Browser Preview Method and System Supporting Multiple Video Formats

Abstract

The present invention discloses a pure browser preview method and system supporting multiple video formats. The preview method includes: using the Emscripten toolchain to compile FFmpeg into an artifact that can run in a browser environment; when the browser does not support the format of a video file, calling the artifact to perform video decoding and format conversion, and outputting the result to the browser; wherein, when the volume of the video file is greater than a preset threshold, performing adaptive dynamic fragmentation on the video file to form a fragmentation queue, and then calling the artifact to sequentially perform video decoding and format conversion on each fragment in the fragmentation queue. The present invention can achieve video playback on a browser without relying on a server, supports video playback functions in multiple formats, and can adaptively and dynamically adjust large video files according to the current resource occupancy rate, improving the conversion efficiency of the video.

Description

Technical Field

[0001] The present invention belongs to the field of browser video playback, and particularly relates to a pure browser preview method and system supporting multiple video formats. Background Art

[0002] In the current field of browser video playback, although there are already multiple libraries (such as Video.js, HLS.js, and Dash.js, etc.) providing video playback support, there are still the following defects: (1) Format support limitations. Most existing solutions rely on HTML5's <video>Labels do not natively support certain video formats (such as AVI, MKV, etc.); (2) Compatibility issues: Different browsers have different levels of support for video formats, posing challenges for developers in achieving multi-platform compatibility; (3) Server dependence: Currently, most technologies rely on the server side for video preview and transcoding. This approach not only increases the server load but also causes latency and bandwidth consumption issues, resulting in an unsatisfactory experience for users when playing videos locally.

[0003] Even if it is possible to directly play videos in multiple video formats on the browser side, video playback often lags for large video files, and there are often problems with video playback delays on devices with poor capabilities, affecting the user's viewing experience. Summary of the Invention

[0004] The purpose of the present invention is to provide a pure browser preview method and system that support multiple video formats to solve the problem of unsmooth video playback proposed in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] On the one hand, the present invention provides a pure browser preview method that supports multiple video formats. The preview method includes

[0007] Using the Emscripten toolchain to compile FFmpeg into an artifact that can run in a browser environment;

[0008] When the browser does not support the format of the video file, calling the artifact for video decoding and format conversion and outputting it to the browser;

[0009] Among them, when the volume of the video file is greater than a preset threshold, the video file is adaptively dynamically segmented to form a segmentation queue, and then the artifact is called to sequentially perform video decoding and format conversion on each segment in the segmentation queue;

[0010] In the adaptive dynamic segmentation, the current segment size is calculated using the current resource occupancy, and the current segment of the video file is obtained based on the current segment size.

[0011] Preferably, the adaptive dynamic segmentation includes:

[0012] Determining the theoretical size of each segment based on a preset number of base segments and the volume of the input video file;

[0013] Adjusting the current segment size according to the current resource occupancy of the device and the theoretical size of the current segment.

[0014] Preferably, the adjustment of the current shard size according to the current resource occupancy of the device and the theoretical size of the current shard includes:

[0015] Calculating the volume adjustment coefficient of the current shard according to the current resource occupancy of the device as:

[0016] In the formula, represents the current memory usage, represents the memory usage threshold, represents the current CPU occupancy rate, represents the CPU occupancy threshold, represents the network bandwidth, represents the network bandwidth threshold;

[0017] Adjusting the size of the current shard by using the volume adjustment coefficient of the current shard and the theoretical size of the current shard as:

[0018]

[0019] In the formula, is the theoretical size of the current shard.

[0020] Preferably, the preview method further includes: while performing adaptive dynamic sharding on a video file that exceeds a preset threshold, detecting key scenes of the video file, and when a key scene is detected, jumping to the corresponding position for priority sharding.

[0021] Preferably, the key scene is characterized by a scene change score. When the scene change score is greater than the preset threshold, there is a key scene at the corresponding position;

[0022] The scene change scoring method includes:

[0023] Calculating an image change score based on the difference in image features between two adjacent frames;

[0024] Calculating the sum of the image change scores between two adjacent frames within a preset time window as the scene change score.

[0025] Preferably, the calculation formula of the image change score is:

[0026]

[0027] In the formula, represents the image index, , represents the total number of images within the preset time window, represents the color difference, represents the motion vector, Indicates texture difference, Indicates inter-frame brightness difference, Indicates edge feature difference, All indicate corresponding weights, Indicates the threshold of color richness difference, Indicates the motion detection threshold, Indicates the texture difference threshold, Indicates the brightness difference threshold, Indicates the edge feature threshold, Indicates the adjustment factor, , Indicates the volume of the video file, Indicates the preset maximum volume of the video file.

[0028] Preferably, in the scene change scoring method, after calculating the image change score, it further includes a correction step of the image change score:

[0029] Obtain the scene type of the inter-frame image through the scene type discrimination strategy;

[0030] Use the scene type discrimination result to correct the corresponding image change score.

[0031] Preferably, the scene type discrimination strategy is: judge whether the scene type score is greater than 1. If so, it is considered that the current scene is a dynamic scene; otherwise, it is a static scene;

[0032] Among them, the calculation formula of the scene type score is:

[0033] Among them, Indicates the motion vector, Indicates the edge feature difference, Indicates the motion detection threshold, Indicates the edge feature threshold;

[0034] The correction formula of the image change score is:

[0035]

[0036] Among them, Is the image change score, Indicates the adjustment factor of the dynamic scene, Indicates the highest threshold for dynamic scene determination, Indicates the adjustment factor of the static scene, Indicates the lowest threshold for static scene determination.

[0037] On the other hand, the invention provides a pure browser preview system that supports multiple video formats. The system includes:

[0038] An FFmpeg decoding artifact that can run independently on a browser for video decoding and format conversion;

[0039] A format judgment module for obtaining the formats supported by the browser and detecting the format supportability of the input video file;

[0040] A sharding processing module for adaptively dynamically sharding video files larger than a preset threshold;

[0041] A video upload module for obtaining the video file input by the user.

[0042] Preferably, the system further includes a key scene detection module for calculating the scene change score of the video file with respect to consecutive frames and performing key scene detection based on the scene change score.

[0043] Compared with the prior art, the beneficial effects of the present invention are

[0044] 1) By directly transplanting the video decoder to the browser side, it supports the playback and decoding of multiple formats, and does not rely on the server for transcoding. Just by loading the corresponding decoding library, various videos can be smoothly played in the browser;

[0045] 2) Through adaptive sharding processing, the size of the current shard is automatically adjusted according to the current resource occupancy, and the most reasonable shard size is allocated for the current resource for format conversion processing, reducing resource waste, optimizing video processing efficiency, and improving the viewing experience;

[0046] 3) Through key scene detection and the priority sharding mechanism triggered under key scenes, a priority viewing function for key scene segments is provided for users. Description of the Drawings

[0047] Figure 1 It is a flowchart of the pure browser preview method of Embodiment 1.

[0048] Figure 2 It is a flowchart of the pure browser preview method of Embodiment 2.

[0049] Figure 3 It is a flowchart of the adaptive dynamic sharding in Embodiment 2.

[0050] Figure 4 It is a schematic diagram of the framework structure of a pure browser preview system supporting multiple video formats of the present invention. Detailed Embodiments

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0052] Referring to Figure 1 As shown, a pure browser preview method supporting multiple video formats in this embodiment includes the pre-operation of the browser and the processing flow of the video.

[0053] The pre-operation of the browser is as follows: By using the Emscripten toolchain, the FFmpeg decoder is compiled into an efficient artifact in bytecode format, enabling it to run independently in the browser environment and improving the video processing speed of the browser.

[0054] Emscripten is a compiler based on LLVM. In theory, it can compile any code that can generate LLVM bitcode into a strict subset of JavaScript, asm.js. In practice, it is mainly used to compile C / C++ code into asm.js. FFmpeg is an open-source software across platforms and can be used for video encoding, decoding, and format conversion. This compilation process ensures the cross-platform compatibility of FFmpeg and optimizes the video decoding and format conversion performance.

[0055] The present invention supports the decoding and playing of multiple video formats, including MP4, AVI, MKV, etc., by transplanting the FFmpeg decoder to the browser side. At the same time, the video decoding and playing are completely completed on the client side, getting rid of the dependence on the server. Even in the case of limited bandwidth or unstable network, smooth playing performance can still be maintained. This artifact can be integrated with the player built into the browser and HTML5 <video>The seamless integration of the tags enables developers to operate through familiar APIs, reducing the learning cost and eliminating the need to re-learn or adjust existing development methods.

[0056] The processing flow of this video includes:

[0057] Step 1, input of the user's video file.

[0058] Step 2, detect the compatibility of the browser with different video processing technologies, obtain the highly efficient execution technology supported by the browser, and use this highly efficient execution technology for subsequent operations. Otherwise, fallback to other compatible processing methods, such as prompting that the browser is not supported or prompting to change the browser.

[0059] Here, the highly efficient execution technology is the call of APIs such as WebAssembly, WebCodecs, and WEBGPU. Different browsers support different underlying highly efficient execution technologies. For example, Firefox supports WebAssembly and WebGPU, partially supports WebCodes, while Safari supports WebAssembly but not WebCodes.

[0060] Step 3, perform a format supportability detection on the video file. If the browser can support the native format or encoding of the video file, the browser directly plays the video file. Otherwise, execute Step 4.

[0061] Step 4, determine whether the volume of the video file is greater than a preset threshold. If so, perform adaptive dynamic sharding on the video file to form a shard queue, and call the highly efficient execution technology supported by the browser to perform multi-threaded video decoding and format conversion on each shard in the shard queue in sequence. Otherwise, directly call the highly efficient execution technology supported by the browser to perform video decoding and format conversion on the video file.

[0062] In Step 4 of the present invention, the size of the video file is judged by its volume. If it is a small file, directly perform video decoding and format conversion. If it is a large file, it is necessary to perform sharding first. During the sharding process, adjust the current shard size of the video according to the current performance of the device and the browser, so that the resource utilization rate at each moment reaches the optimal value, thereby making the format conversion speed of the video segment under this optimal resource utilization rate also reach the optimal value, to improve the video conversion performance. At the same time, perform multi-threaded simultaneous conversion processing on the shards to ensure that the format or encoding is completed in a very short time, reducing the user's waiting time and improving the loading speed.

[0063] In this adaptive dynamic sharding, first set the threshold of the number of shards, and then dynamically calculate the size of the current shard based on this threshold and the current resource occupancy, ensuring that the total number of shards is close to the shard number threshold while being able to dynamically adjust the volume of each shard according to the resource occupancy to optimize the sharding processing efficiency. The specific process is as follows:

[0064] Step 4.1, based on the preset basic number of shards and the volume of the input video file determine that the theoretical size of each shard is:

[0065] Step 4.2, calculate the volume adjustment coefficient of the current shard according to the current resource occupancy as:

[0066]

[0067] In the formula, represents the current memory usage, represents the memory usage threshold, represents the remaining memory ratio, represents the current CPU occupancy rate, represents the CPU occupancy threshold, represents the remaining CPU ratio, represents the network bandwidth, represents the network bandwidth threshold, represents the network bandwidth ratio, ;

[0068] Step 4.3, adjust the size of the current shard using the volume adjustment coefficient of the current shard and the theoretical size of the current shard as:

[0069] .

[0070] Since each shard requires a separate file handle, too many shards will cause a large amount of occupation, and the processing of each shard requires the allocation of a certain amount of memory. Too many shards will occupy memory resources and exacerbate memory fragmentation. In the present invention, the theoretical size of each shard calculated through Step 4.1 is the same, and for different shards, dynamic fine-tuning is performed based on the same volume as the benchmark to ensure the reliability of the shards and prevent the number of shards far exceeding the threshold from occurring under the poor device capabilities of users. In this adaptive dynamic algorithm, when resources are sufficient, is larger, and the shard size is closer to the base value; when resources are tight, is smaller, and the shard size will decrease.

[0071] In step 4.2 of the present invention, by using memory, network bandwidth, and CPU occupancy rate as calculation parameters for resource occupancy, rather than using only single parameters such as memory and CPU occupancy rate to represent resource occupancy, potential performance bottlenecks can be better identified, ensuring the accuracy of the representation of resource occupancy; by jointly calculating the adjustment coefficient of the shard volume using memory, network bandwidth, and CPU occupancy rate, when a certain resource is overloaded, measures such as reducing the number of shards or adjusting the shard size can be taken to relieve the pressure on the bottleneck resource, avoiding the risk of performance degradation or system crash, enhancing the scalability of the system, and enabling the system to dynamically adjust shards according to the usage of resources when the external load changes, ensuring reasonable allocation and use of resources.

[0072] Step 5: Generate a Blob URL to return the converted streaming media data in real time for the user to play immediately on the browser.

[0073] A Blob URL is a special URL generated by the browser for temporarily accessing binary data (such as pictures, videos, etc.) stored in memory. After directly assigning the blob URL to the video tag or the player module, the browser can play the video; the generation method of this Blob URL is common knowledge in the art and will not be elaborated here.

[0074] In step 5 of the present invention, during the process of sharding and converting the video file, the user can achieve real-time streaming playback of the video through the generated Blob URL. This design allows the user to watch the video while the video is being converted, greatly improving the playback response speed.

[0075] Step 6: Merge all the shards after format conversion to generate a new video file for the user to download or play directly on the browser.

[0076] In the present invention, whether the browser can support the native format or encoding of the video file, or after the video file is decoded and format-converted through the workpiece, the browser will use the efficient execution technology detected in step 2 for video playback. The modular-designed workpiece can use different underlying technologies for acceleration for different browsers, either relying on the native functions of the browser or using efficient compilation modules to enhance the overall processing ability.

[0077] Refer to Figure 2 As shown, the above pure browser preview method is improved: in step 4, while performing adaptive dynamic sharding on the video file exceeding the preset threshold, key scene detection is performed on the video file, and when a key scene is detected, it jumps to the corresponding position for priority sharding.

[0078] In the present invention, when a key scene is detected, it is necessary to determine whether the detected key scene has been fragmented. If it has been fragmented, no secondary fragmentation process is performed. Otherwise, it jumps to the corresponding position of the key scene for priority fragmentation. After the priority fragmentation process of the key scene is completed, it still needs to return to the previous fragmentation task, and waits for the appearance of a new key scene for the next jump.

[0079] In this embodiment, the key scene is characterized by a scene change score. When the scene change score is greater than a preset threshold, it indicates that the video segment corresponding to the scene change score is a key scene.

[0080] In step 4 of the present invention, each time a fragmentation is calculated and processed, the fragmentation queue is updated. Refer to Figure 3 As shown, for example, when thread A adaptively dynamically fragments a video file in chronological order to obtain fragments 1 to 11, thread B detects that the scene change score of the video file from 12:05 to 12:30 is greater than the preset threshold. At this time, 12:05 is marked. Thread A determines that the starting point of the current fragment is at the 20MB position of the video according to this marked point, takes it as the starting point of the current fragment, and after calculating the size of the current fragment through steps 4.1 - 4.3, performs fragmentation processing to obtain fragment 12, and then adaptively dynamically fragments in chronological order. It should be noted that in subsequent adaptive dynamic fragmentations, no fragmentation processing is performed on the video segment corresponding to this fragment 12.

[0081] In the present invention, by setting a priority fragmentation system for key scenes, it can ensure that users can play immediately for key scene segments. Through the instant playback function of this key video segment, a new viewing option is provided for users, improving the user experience.

[0082] As a specific implementation manner of this scene change score, its specific process is as follows:

[0083] (a) Calculate the image change score based on the image feature difference between two adjacent frames , represents the image frame index;

[0084] (b) Calculate the sum of the image change scores between two adjacent frames within a preset time window as the scene change score, denoted as .

[0085] Here, the similarity between two adjacent image frames is calculated based on the differences in image features, and then the sum of the similarities of consecutive frames is used to characterize the scene change score of the video segment. The higher the scene change score, the more information is covered in the corresponding video segment, and at this time, it is considered that the video segment meets the user's requirements. The preset time window is used to fix the number of image change scores when calculating the scene change score. This time window is a sliding time window, and the moving step size can be set according to the actual situation.

[0086] Here, the image change score is calculated by the following formula:

[0087]

[0088] In the formula, represents the image index, , represents the total number of images under the preset time window, represents the color difference, represents the motion vector, represents the texture difference, represents the inter-frame luminance difference, represents the edge feature difference, all represent the corresponding weights, represents the color richness difference threshold, represents the motion detection threshold, represents the texture difference threshold, represents the luminance difference threshold, represents the edge feature threshold, represents the adjustment factor, , represents the video file size, represents the preset maximum video file size; the acquisition methods of the above parameters are conventional technical means in the art and will not be elaborated here.

[0089] In this embodiment, the scene change score is calculated by combining visual features such as color, motion, texture, luminance, and edge, ensuring the reliability of the scene change score, and introducing weight coefficients that can be dynamically adjusted according to user needs to meet different user requirements and improve the user's viewing experience.

[0090] In this embodiment, by setting the adjustment factor, the scoring deviation caused by different video file sizes is avoided, helping the algorithm to better adapt to video inputs of different qualities and sizes, ensuring accurate detection of scene changes under different conditions, and avoiding misjudgment and performance degradation caused by over-sensitivity when processing large files, which is important for complex and variable video content and helps to improve the overall performance of scene change detection.

[0091] Further, optimize the scene change score in this embodiment: after obtaining the image change score, determine dynamic and static scenes through the scene type discrimination strategy to obtain the scene determination result, and use the score correction strategy in different scenes to correct the image change score. The corrected image change score is used in the calculation of the scene change score.

[0092] In this scene type discrimination strategy, if the scene type score is greater than 1, it is considered that there is a dynamic change between the current frame image and the next frame image, which is a dynamic scene; otherwise, it is a static scene. The calculation of the scene type score is shown in the following formula:

[0093]

[0094] In the formula, represents the motion vector, represents the edge feature difference, represents the motion detection threshold, represents the edge feature threshold.

[0095] Here, by standardizing the changes in motion and edge features, the degree of change in different video contents and environments can be fairly compared to determine the scene type.

[0096] The image change score correction strategy in different scenes is shown in the following formula:

[0097]

[0098] Among them, is the image change score, represents the adjustment factor for the dynamic scene, represents the highest threshold for dynamic scene determination, represents the adjustment factor for the static scene, represents the lowest threshold for static scene determination.

[0099] Here, the original score is adjusted by the scene type to more accurately reflect the degree of scene change; in the dynamic scene, the score will be amplified to more sensitively detect changes; in the static scene, the score will be reduced to avoid misjudging as a scene change due to minor changes.

[0100] Referring to Figure 4 as shown, the present invention also provides a pure browser preview system supporting multiple video formats, including:

[0101] A video upload module for obtaining the video file input by the user;

[0102] The format judgment module is used to obtain the video formats supported by the browser and perform format support detection on the video file input by the user obtained by the video upload module;

[0103] The sharding processing module is used to perform adaptive dynamic sharding on the video file when the volume of the video file input by the user is greater than the preset threshold and the browser does not support the format of the video file input by the user;

[0104] The key scene detection module is used to calculate the scene change score of the video file with respect to consecutive frames and perform key scene detection based on the scene change score;

[0105] The triggering mechanism triggers the sharding processing module to jump to the corresponding position for priority processing of sharding when a key scene is detected;

[0106] The FFmpeg decoding component can run independently on the browser and is used for decoding and format conversion of video files or shards.

[0107] In this format judgment module, when the browser supports the format of the video file input by the user, the browser directly plays the video. The format judgment module is respectively connected to the FFmpeg decoding component and the sharding processing module. At the same time, the sharding processing module is connected to the FFmpeg decoding component. When the video file is too large, it is first sharded and then format-converted. When the video file is not large, it is directly format-converted and then directly played by the browser.

[0108] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / video> < / video>

Claims

1. A browser-only preview method supporting multiple video formats, characterized in that: The preview method includes Use the Emscripten toolchain to compile FFmpeg into an artifact that can run in a browser environment; When the browser does not support the format of the video file, call the artifact to decode and convert the video format and output it to the browser; Among them, when the size of the video file is larger than the preset threshold, the video file is adaptively and dynamically segmented, and after a segment queue is formed, the workpiece is called to sequentially perform video decoding and format conversion on each segment in the segment queue; in the adaptive dynamic segmentation, the current resource occupancy is used to calculate the current segment size, and the current segment of the video file is obtained based on the current segment size; The preview method further includes: while adaptively and dynamically slicing the video file exceeding a preset threshold, detecting key scenes on the video file, and jumping to a corresponding position for priority slicing when a key scene is detected; the key scene is represented by a scene change score, and when the scene change score is greater than a preset threshold, a key scene exists at the corresponding position; Scene change scoring methods include: Calculate the image change score based on the difference in image features between two adjacent frames; Calculate the sum of the image change scores between two adjacent frames in a preset time window as the scene change score; The calculation formula for the image change score is: Where j represents the image index, j = 1, 2, …, J-1, J represents the total number of images in the preset time window, ΔC represents the color difference, ΔM represents the motion vector, ΔT represents the texture difference, ΔL represents the brightness difference between frames, ΔE represents the edge feature difference, τ, β, γ, δ,∈ all represent the corresponding weights, T color represents the color richness difference threshold, T motion represents the motion detection threshold, T texture represents the texture difference threshold, T brightness represents the brightness difference threshold, T edge Indicates edge feature threshold, size factor Indicates the adjustment factor, size factor =F / F max ,F represents the size of the video file, F max Indicates the preset maximum video file size.

2. A browser-only preview method supporting multiple video formats as claimed in claim 1, characterized in that: The adaptive dynamic slicing comprises: Determine the theoretical size of each segment based on the preset number of basic segments and the volume of the input video file; Adjust the current shard size based on the current resource usage of the device and the theoretical size of the current shard.

3. A browser-only preview method supporting multiple video formats as claimed in claim 2, characterized in that: The adjusting of the current slice size according to the current resource occupancy of the device and the theoretical size of the current slice includes: The volume adjustment coefficient α of the current shard is calculated based on the current resource usage of the device: In the formula, M represents the current memory usage, T m represents the memory usage threshold, C represents the current CPU usage, T C represents the CPU usage threshold, N represents the network bandwidth, T n Indicates the network bandwidth threshold; Adjust the size of the current shard using the volume adjustment coefficient of the current shard and the theoretical size of the current shard Chunk size for: Chunk size =Basechunk_size×α Where Basechunk_size is the theoretical size of the current shard.

4. A browser-only preview method supporting multiple video formats as claimed in claim 1, characterized in that: The scene change scoring method further includes, after calculating the image change score, a correction step of the image change score: obtaining the scene type of the inter-frame image by a scene type discrimination strategy; The scene type discrimination result is used to correct the corresponding image change score.

5. A browser-only preview method supporting multiple video formats as claimed in claim 4, characterized in that: The scene type discrimination strategy is: determine whether the scene type score is greater than 1, if so, the current scene is considered to be a dynamic scene, otherwise it is a static scene; The calculation formula for the scene type score is: Among them, ΔM represents the motion vector, ΔE represents the edge feature difference, and T motion represents the motion detection threshold, T edge Indicates the edge feature threshold; The correction formula of the image change score is: Among them, S is the image change score, K dynarmic Represents the adjustment factor of dynamic scenes, T dynarmic Indicates the highest threshold for dynamic scene judgment, K static represents the adjustment factor for static scenes, T static Indicates the minimum threshold for static scene judgment.

6. A browser-only preview system supporting multiple video formats, characterized in that: Using the pure browser preview method according to any one of claims 1 to 5, the system comprises: FFmpeg decoding artifact, which can run independently on the browser for video decoding and format conversion; The format determination module is used to obtain the formats supported by the browser and perform format support detection on the input video file; A fragmentation processing module, used for adaptive dynamic fragmentation processing of video files larger than a preset threshold; The video upload module is used to obtain the video files input by the user.

7. A browser-only preview system supporting multiple video formats as claimed in claim 6, characterized in that: The system also includes a key scene detection module, which is used to calculate scene change scores of continuous frames in the video file and perform key scene detection based on the scene change scores.

Images