An intelligent processing method, system, device and medium for video merging

Abstract

The application discloses an intelligent processing method, system and device for video merging and medium, and specifically comprises the following steps: dynamically dividing a to-be-processed video file into multiple variable segments; constructing a Merkle check tree according to the unique hash values of all the variable segments; obtaining first video data after segment transmission, extracting a visual feature vector of the first video data, identifying a subject object in a picture, performing three-level classification on a scene type and an action category, and generating second video data; performing frame position identification, key frame caching and time sequence alignment of multiple video segments on the second video data to form a merged video stream; eliminating joint traces of the merged video stream, monitoring video quality indexes in real time, dynamically adjusting quantization parameters of the merged video stream until the merged video stream reaches an output standard. The application improves the processing efficiency of video merging, greatly shortens the time consumption of large file merging, and meets the needs of users for efficient video merging.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an intelligent processing method, system, device and medium for video merging. Background Technology

[0002] In the current field of video processing technology, video merging is a common requirement. Whether it's individual users creating home videos or professional video editors performing post-production work for films and television shows, multiple video clips need to be merged into a single, complete video. However, current video merging tools have many shortcomings, severely impacting user experience and merging efficiency.

[0003] First, operational complexity is a major drawback of existing video merging tools. Professional video editing software such as Premiere and Final Cut Pro, while powerful, require users to master complex non-linear editing techniques, resulting in a high learning curve, typically exceeding 200 hours of study time to become proficient. This is undoubtedly a significant hurdle for ordinary users, making it difficult for them to easily complete simple tasks such as video merging. Multimedia technology occupies a central position in today's society, and its widespread application in film and television production, advertising design, and education and training further highlights the importance of video processing needs. However, current mainstream video merging tools, such as Adobe Premiere and Final Cut Pro, are complex to operate and have a high learning curve, making them difficult for non-professional users to use and failing to meet the practical needs for speed and convenience.

[0004] Secondly, poor cross-platform compatibility is a significant problem with existing video merging tools. Different operating systems require dedicated encoders, which not only increases the difficulty for users but may also lead to a 30%-50% quality loss during format conversion. This quality loss is unacceptable for users seeking high-definition video quality. Furthermore, the wide variety of tools available for different operating systems, with significant differences in experience between paid and free versions, further exacerbates user confusion and operational inefficiency.

[0005] Finally, low processing efficiency is a major pain point of existing video merging tools. Traditional video merging methods typically employ a full retransmission mechanism, meaning the entire video file needs to be retransmitted for each merge, leading to an exponential increase in time consumption for merging large files. For users who frequently need to merge large files, this inefficient approach is undoubtedly a nightmare. The limitations of existing methods in terms of operational efficiency and user experience are particularly prominent. Complex operation processes and non-intuitive interface designs require users to invest significant time in learning, while the diverse tool ecosystem deters ordinary users. These shortcomings not only reduce the efficiency of video merging but also limit the widespread adoption of the technology among a broader population. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent processing method, system, device, and medium for video merging, which improves the processing efficiency of video merging, greatly reduces the time required for merging large files, meets users' needs for efficient video merging, and solves at least one of the aforementioned problems in the prior art.

[0007] In a first aspect, the present invention provides an intelligent video merging processing method, the method specifically comprising: The video file to be processed is obtained, and the Rabin fingerprinting algorithm is used to dynamically divide the video file into multiple variable segments, generating a unique hash value for each variable segment. A Merkle check tree is constructed based on the unique hash value of all variable fragments, and fragmented transmission is performed via the WebRTC protocol; The first video data after being fragmented and transmitted is obtained. The visual feature vector of the first video data is extracted by the ResNet50 deep neural network. The YOLOv8-tiny object detection model is combined to identify the main object in the picture. The CLIP image classification model is linked to perform three-level classification of scene type and action category to generate the second video data. Based on the second video data, a video segment association graph is constructed. Based on the video segment association graph, the frame position of each video segment is identified, key frame is cached, and multiple video segments are time-series aligned to form a merged video stream. The pyramid fusion algorithm is used to eliminate the seams in the merged video streams, and the VMAF evaluation model is used to monitor video quality indicators in real time and dynamically adjust the quantization parameters of the merged video streams until the merged video streams meet the output standards.

[0008] Secondly, the present invention provides an intelligent video merging processing system, the system specifically comprising: The first processing module is used to obtain the video file to be processed, and uses the Rabin fingerprinting algorithm to dynamically divide the video file to be processed into multiple variable segments, and generates a unique hash value for each variable segment. The second processing module is used to construct a Merkle check tree based on the unique hash value of all variable fragments and transmit the fragments via the WebRTC protocol; The third processing module is used to acquire the first video data after the segmented transmission, extract the visual feature vector of the first video data through the ResNet50 deep neural network, identify the main object in the picture by combining the YOLOv8-tiny object detection model, and link the CLIP image classification model to perform three-level classification of scene type and action category to generate the second video data. The fourth processing module is used to construct a video segment association map based on the second video data. Based on the video segment association map, it performs frame position identification, key frame caching, and temporal alignment of multiple video segments on the second video data to form a merged video stream. The fifth processing module is used to eliminate seam traces in the merged video streams using the pyramid fusion algorithm, and to monitor video quality indicators in real time using the VMAF evaluation model, dynamically adjusting the quantization parameters of the merged video streams until the merged video streams meet the output standards.

[0009] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements an intelligent processing method for video merging as described in any of the above methods.

[0010] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements an intelligent processing method for video merging as described in any of the above methods.

[0011] Compared with the prior art, the present invention has at least one of the following technical effects: 1. This invention improves the processing efficiency of video merging, greatly shortens the time required for merging large files, and meets users' needs for efficient video merging.

[0012] 2. This invention achieves efficient and secure video segmentation and transmission through Rabin fingerprinting, Merkle check tree, and WebRTC transmission, thereby improving the flexibility and reliability of video merging.

[0013] 3. The present invention dynamically segments the video into variable segments and generates unique hash values ​​to ensure the accuracy and uniqueness of the segments, which facilitates subsequent processing and verification.

[0014] 4. This invention constructs a Merkle check tree and optimizes fragmented transmission through the WebRTC protocol, thereby improving transmission efficiency and security, while implementing a dynamic scheduling strategy to adapt to network changes.

[0015] 5. This invention combines deep neural networks and object detection models to accurately extract video features and classify scenes and actions, thereby improving the ability to understand and process video content.

[0016] 6. This invention constructs a video segment association graph, which effectively represents the similarity and temporal relationship between video segments, providing accurate association information for video merging.

[0017] 7. This invention improves the quality of video merging by ensuring the continuity and smoothness of merged video streams through frame position recognition, key frame caching, and timing alignment.

[0018] 8. This invention uses a pyramid fusion algorithm to eliminate seam traces and uses a VMAF model to monitor and adjust video quality in real time, achieving high-quality video merging output. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an intelligent video merging processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an intelligent video merging processing system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0023] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0025] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0027] In this application embodiment, the entity executing the process includes a terminal device. This terminal device includes, but is not limited to, devices capable of executing the methods disclosed in this application, such as servers, computers, smartphones, and tablets. Figure 1 A flowchart illustrating the intelligent video merging processing method disclosed in the first embodiment of the present invention is shown below, and is described in detail below: S101: Obtain the video file to be processed, and use the Rabin fingerprinting algorithm to dynamically divide the video file to be processed into multiple variable segments, generating a unique hash value for each variable segment.

[0028] In this embodiment, dividing a large video file into multiple smaller blocks during video processing, transmission, and storage can improve processing efficiency, transmission speed, and storage flexibility. Traditional fixed-size block methods cannot dynamically adjust the block size according to changes in video content, while the Rabin fingerprint block algorithm can dynamically determine the block boundaries based on changes in data content, resulting in more reasonable block division.

[0029] Specifically, it reads the video file to be processed. This file can be of any format, such as MP4 or AVI. The reading method can be byte-by-byte or block-by-block, depending on the file size and system memory limitations.

[0030] Configure the parameters of the Rabin fingerprint segmentation algorithm, including the sliding window size, minimum block size, and maximum block size. The sliding window size is typically chosen to be a small number of bytes, such as 16 bytes, for fast fingerprint value calculation. The minimum and maximum block sizes can be set according to actual needs to control the range of block sizes.

[0031] Starting from the beginning of the video file, an initial fingerprint value is calculated from the data within a sliding window using a hash function. The sliding window is then moved one byte to the right, and the fingerprint value is quickly updated using specific mathematical operations (such as bitwise XOR, bitwise shift operations, etc.) instead of recalculating the hash value of the entire window. The new fingerprint value is checked to see if it meets a specific condition (e.g., the result of a bitwise AND operation between the fingerprint value and a preset mask is 0). If the fingerprint value meets the condition, the starting position of the current sliding window is used as a block boundary, and the preceding data is treated as a fragment. If the sliding window moves a certain number of bytes (exceeding the maximum block size limit) and still does not find a fingerprint value that meets the condition, a block is forced at the current position to avoid creating excessively large blocks. For each variable fragment, a hash value is calculated for its content using a hash function (such as SHA-256). The calculated hash value is used as a unique identifier for that fragment, for subsequent data verification, deduplication, or distributed storage.

[0032] In this embodiment, the Rabin fingerprinting algorithm can quickly and dynamically divide video files into multiple variable segments, improving video processing efficiency. The segment size is dynamically adjusted according to changes in video content, resulting in more reasonable segmentation and adapting to the characteristics and needs of different video files. A unique hash value is generated for each segment, facilitating subsequent data verification, deduplication, and distributed storage, thus improving data reliability and storage efficiency.

[0033] S102, construct a Merkle check tree based on the unique hash value of all variable fragments, and transmit the fragments via the WebRTC protocol.

[0034] In this embodiment, ensuring data integrity and reliability is crucial during video transmission. Traditional transmission methods often rely on retransmission mechanisms to handle data loss or corruption, but this increases transmission latency and bandwidth consumption. Merkle checksums are an effective data verification structure that can quickly verify data integrity. Meanwhile, the WebRTC protocol, as a real-time communication protocol, features low latency and high efficiency, making it suitable for transmitting real-time data such as video.

[0035] Specifically, the video file is dynamically divided into multiple variable segments using the Rabin fingerprinting algorithm or other segmentation methods, and a unique hash value is generated for each segment. The unique hash values ​​of all variable segments are used as leaf nodes of the Merkle checksum tree. A hash operation (such as using the SHA-256 algorithm) is performed on the hash values ​​of adjacent leaf nodes to obtain the hash value of the parent node. This process is repeated layer by layer upwards until the root hash value of the Merkle checksum tree is obtained.

[0036] Establish a WebRTC connection to ensure real-time communication between the transmitting and receiving parties. Send video fragments and their corresponding hash values ​​(or simply transmit the fragment sequence number and root hash value, with the receiver requesting the specific fragment based on the sequence number) to the receiving party via the WebRTC protocol. Upon receiving a fragment, the receiving party calculates its hash value and compares it with the hash value provided by the sender or verified through a Merkle checksum tree to verify the fragment's integrity.

[0037] For example, suppose a video file is divided into N fragments, with fragment hash values ​​H1, H2, ..., HN. H1, H2, ..., HN are used as leaf nodes of a Merkle checksum tree. SHA-256 operations are performed on the hash values ​​of adjacent leaf nodes to obtain the hash value of the first-level parent node. For example, the operation on H1 and H2 yields the parent node hash value P1, the operation on H3 and H4 yields the parent node hash value P2, and so on. The same operation is performed on the hash values ​​of the first-level parent nodes to obtain the hash values ​​of the second-level parent nodes, until the root hash value R of the Merkle checksum tree is finally obtained. The sender and receiver establish a WebRTC connection. The sender sends the video fragment (or fragment sequence number and root hash value R) to the receiver via the WebRTC protocol. After receiving the fragment, the receiver calculates its hash value and compares it with the hash value provided by the sender or the hash value obtained through the Merkle checksum tree (verified using the received fragment hash value and root hash value R). If the hash values ​​match, the fragment is complete and undamaged; if they do not match, the fragment may have been lost or damaged during transmission, and the receiver can request the sender to retransmit the fragment.

[0038] In this embodiment, the receiver can quickly verify the integrity of video fragments using the Merkle check tree, without waiting for all fragments to be received before verification, thus improving transmission efficiency. The WebRTC protocol is characterized by low latency and high efficiency; combined with the Merkle check tree mechanism, it ensures the integrity and reliability of video fragments during transmission. Because the Merkle check tree can quickly locate lost or corrupted fragments, there is no need to transmit the entire file or a large amount of redundant data for verification and retransmission, reducing bandwidth consumption. The WebRTC protocol is suitable for real-time communication scenarios, such as video conferencing and live streaming; combined with the Merkle check tree mechanism, it can provide users with a smooth and stable video transmission experience.

[0039] S103: Obtain the first video data after segmented transmission, extract the visual feature vector of the first video data through the ResNet50 deep neural network, identify the main object in the picture by combining the YOLOv8-tiny object detection model, and link the CLIP image classification model to perform three-level classification of scene type and action category to generate the second video data.

[0040] In this embodiment, accurate and efficient identification and classification of video content is crucial during video processing and analysis. Traditional video processing methods often rely on manually designed feature extraction algorithms, which have limited generalization ability and accuracy. With the development of deep learning technology, deep neural networks have demonstrated powerful performance in video processing and analysis.

[0041] Specifically, the first segmented video data is acquired via network transmission or other means. Each frame of the first video data is then input into a pre-trained ResNet50 deep neural network. The ResNet50 network performs deep feature extraction on the input images, outputting a visual feature vector for each image. These feature vectors contain high-level semantic information about the images, providing a foundation for subsequent object detection and image classification.

[0042] The visual feature vectors extracted by ResNet50 are input into the YOLOv8-tiny object detection model. The YOLOv8-tiny model performs fast and accurate object detection on the input feature vectors, identifying the main objects in the image and outputting the object's category and location information. The main object region identified by YOLOv8-tiny is cropped out and used as input to the CLIP image classification model. The CLIP model combines text and image information to perform three-level classification of the cropped main object region based on scene type and action category. First, it identifies the broad category of the scene (e.g., indoor, outdoor); second, it identifies the specific scene type (e.g., home, office, street); finally, it identifies the action category in the scene (e.g., walking, running, sitting). Based on the output results of the YOLOv8-tiny and CLIP models, the first video data is labeled or processed to generate second video data containing the main object category, location information, and scene type and action category classification results.

[0043] In this embodiment, by combining advanced models such as ResNet50, YOLOv8-tiny, and CLIP, this invention can accurately identify the main objects in video footage and perform detailed classification of scene types and action categories, thereby improving the accuracy of video content recognition. The YOLOv8-tiny model is lightweight and highly efficient, enabling rapid object detection. Meanwhile, the CLIP model combines text and image information for classification, improving processing efficiency and generalization ability. This invention is applicable to various video processing and analysis scenarios, such as video surveillance, video editing, and video recommendation, providing accurate and efficient video content recognition and processing solutions for these scenarios.

[0044] S104. Construct a video segment association map based on the second video data. Based on the video segment association map, identify the frame position of each video segment, cache key frames, and align the temporal sequence of multiple video segments to form a merged video stream.

[0045] In this embodiment, video processing and analysis often require merging multiple video segments to form a complete video stream. However, since video segments may originate from different sources and have different timing sequences and frame rates, directly merging these segments often leads to problems such as timing discrepancies, frame loss, or duplication. To address these issues, this invention provides a method for constructing a video segment association graph based on second video data and processing and merging video data based on this graph, aiming to improve the accuracy and efficiency of video merging.

[0046] Specifically, the second video data is analyzed to extract feature information for each video segment, such as frame content, temporal sequence information, and frame rate. Based on the extracted feature information, a video segment association graph is constructed. Nodes in the graph represent video segments, and edges represent the relationships between segments, such as temporal order and content similarity. Based on the video segment association graph, the frame positions of each video segment in the merged video stream are identified. By analyzing the edge and node information in the graph, the starting and ending frames of each video segment in the merged video stream are determined.

[0047] After identifying the frame positions of each video segment, keyframes in the video segments are cached. Keyframes are representative frames in a video segment that reflect the main content of the segment. Cached keyframes improve the efficiency and accuracy of video merging. Based on the video segment association map and frame position identification results, multiple video segments are time-aligned. By adjusting the playback order and frame rate of the video segments, the correct temporal relationship between the segments is maintained in the merged video stream. The video segments that have undergone frame position identification, keyframe caching, and time alignment are merged to form a complete merged video stream.

[0048] In this embodiment, by constructing a video segment association graph and performing frame position identification, keyframe caching, and timing alignment based on the graph, the present invention can accurately merge multiple video segments into a complete video stream, avoiding problems such as timing discrepancies, frame loss, or duplication. By caching keyframes and performing timing alignment based on the graph, the present invention can reduce the computational load during the video merging process and improve merging efficiency. The present invention can handle video segments from different sources with different timing sequences and frame rates, providing greater flexibility for video processing.

[0049] S105 uses a pyramid fusion algorithm to eliminate seams in the merged video streams and uses a VMAF evaluation model to monitor video quality indicators in real time, dynamically adjusting the quantization parameters of the merged video streams until the merged video streams meet the output standards.

[0050] In this embodiment, during the video merging process, due to potential differences in content, lighting, and color between different video segments, the merged video stream often exhibits noticeable seams, affecting the viewing experience. Furthermore, video quality directly impacts user satisfaction; therefore, it is necessary to conduct quality assessments and optimizations on the merged video stream.

[0051] Specifically, the seam regions in the merged video streams are identified. A pyramid fusion algorithm is used to perform multi-scale, multi-directional fusion processing on these seam regions to eliminate seam artifacts. The pyramid fusion algorithm constructs a multi-scale image pyramid to fuse images of different scales, achieving a smooth transition effect. The VMAF evaluation model is used to monitor the merged video stream in real time, obtaining video quality indicators such as sharpness, color fidelity, and motion smoothness. The VMAF evaluation model is a fusion model that integrates multiple video quality assessment methods, more accurately reflecting the overall video quality. Based on the video quality indicators monitored by the VMAF evaluation model, the quantization parameters of the merged video stream, such as bitrate, frame rate, and encoding format, are dynamically adjusted. By adjusting the quantization parameters, the video quality is optimized until the merged video stream meets the output standard.

[0052] In this embodiment, a pyramid fusion algorithm is used to perform multi-scale, multi-directional fusion processing on the seam area of ​​the merged video streams, effectively eliminating seam traces and improving the viewing experience. Real-time monitoring of the merged video stream using the VMAF evaluation model accurately obtains video quality indicators, providing a strong basis for video quality optimization. Based on the monitoring results of the VMAF evaluation model, the quantization parameters of the merged video stream are dynamically adjusted, optimizing video quality and ensuring the merged video stream meets output standards.

[0053] In some embodiments, step S101 above, which involves dynamically dividing the video file to be processed into multiple variable segments using the Rabin fingerprinting algorithm and generating a unique hash value for each variable segment, specifically includes: The video file to be processed is scanned byte by byte through a sliding window, and the fingerprint value of each sliding window is calculated using the Rabin fingerprint algorithm. When the fingerprint value of any sliding window is greater than the preset fingerprint threshold n times consecutively, a temporary fragmentation boundary marker is triggered, and the start code of the first I-frame after the temporary fragmentation boundary marker appears is set as the final fragmentation cutting point. The video file to be processed is divided into multiple variable segments based on the average compression efficiency by dynamically selecting the segment size threshold and the final segment cutting pad and segment size threshold. Calculate the full SHA3-256 hash value of each variable segment and use the full SHA3-256 hash value as the unique hash value of the variable segment.

[0054] In this embodiment, dividing a video file into multiple segments during video processing, storage, and transmission can improve processing efficiency and reduce storage and transmission costs. However, traditional fixed-segmentation methods cannot dynamically adjust segment sizes according to changes in video content, resulting in low segmentation efficiency. To address this issue, this invention provides a method that uses the Rabin fingerprinting algorithm to dynamically segment the video file to be processed into multiple variable segments and generates a unique hash value for each variable segment, aiming to improve the efficiency and flexibility of video file segmentation.

[0055] Specifically, the sliding window size is set to 1024 bytes, and the video file to be processed is scanned byte by byte. The Rabin fingerprint algorithm is used to calculate the fingerprint value of each sliding window. The Rabin fingerprint algorithm is a multinomial-based hash algorithm that can effectively detect changes in data blocks. A preset fingerprint threshold T is set. When the fingerprint value of a sliding window is greater than T for n consecutive times (e.g., n=5), a temporary segment boundary marker is triggered. This indicates that a significant change has occurred in the video content at this point, possibly a scene change or the occurrence of an important event.

[0056] After the temporary fragment boundary marker appears, the video file continues to be scanned to find the first I-frame start code. The I-frame is a key frame in video encoding and can be decoded independently. Therefore, setting the I-frame start code as the final fragment cutting point can ensure the integrity of the fragments.

[0057] Based on historical data or experimental results, the relationship between average compression efficiency and fragment size is determined. A fragment size threshold S is dynamically selected based on the current video file's compression efficiency. Higher compression efficiency allows for a larger fragment size threshold. The video file to be processed is divided into multiple variable fragments based on the final fragment cutting point and the fragment size threshold S. The fragment size fluctuates around S and is dynamically adjusted according to changes in the video content. For each variable fragment, its full SHA3-256 hash value is calculated. SHA3-256 is a secure hash algorithm that generates a 256-bit hash value with extremely high uniqueness and collision resistance. The full SHA3-256 hash value is used as the unique hash value for the variable fragment, for subsequent video file management, retrieval, and verification.

[0058] In this embodiment, the Rabin fingerprint segmentation algorithm is used to dynamically adjust the segment size and segment the video based on changes in video content, improving segmentation efficiency. Setting the start code of the I-frame as the final segmentation point ensures segment integrity, allowing each segment to be decoded independently. The SHA3-256 algorithm is used to generate unique hash values ​​for variable segments, enhancing segment uniqueness and collision resistance, facilitating subsequent video file management. The segment size threshold is dynamically selected based on average compression efficiency, allowing the segment size to be adjusted according to the characteristics of the video file, improving segmentation flexibility.

[0059] In some embodiments, step S102 above, which involves constructing a Merkle check tree based on the unique hash values ​​of all variable fragments and transmitting the fragments via the WebRTC protocol, specifically includes: Use the first k bits of the unique hash value of each variable partition as a lightweight identifier, and set the leaf nodes according to the lightweight identifier; Based on the leaf nodes, the parent node is formed by recursively calculating the connection digest of the hash values ​​of adjacent nodes until a complete binary tree structure is generated, thus obtaining the Merkle check tree. Each variable fragment, along with its corresponding fragment index, lightweight identifier value, and verification path from the leaf node to the root node, is encapsulated into a transmission data packet. The transmission priority weight is calculated based on the proximity and urgency of the fragment index to determine the fragment transmission sequence, while network latency is monitored and the scheduling strategy is adjusted in real time. A point-to-point transmission channel is established using the WebRTC protocol, and data packets are sent according to the fragmented transmission sequence, while the ICE candidate address is continuously updated during transmission.

[0060] In this embodiment, ensuring data integrity and security is crucial during video transmission. Traditional transmission methods often rely on retransmission mechanisms to handle data loss or corruption, but this increases transmission latency and bandwidth consumption. Furthermore, efficient transmission of video fragments also requires consideration of factors such as network conditions and task urgency. To address these issues, this invention provides a method for constructing a Merkle checksum tree based on the unique hash values ​​of all variable fragments and transmitting fragments via the WebRTC protocol, aiming to improve the integrity and efficiency of video transmission.

[0061] Specifically, for each variable fragment, the first k bits of its unique hash value (e.g., k=8) are taken as a lightweight identifier. The leaf nodes of the Merkle checksum tree are then set based on these lightweight identifiers. Each leaf node corresponds to a variable fragment. Starting from the leaf node, parent nodes are formed by recursively calculating the link digests of the hash values ​​of adjacent nodes (e.g., using the SHA256 algorithm). This recursive process continues until a complete binary tree structure is generated, resulting in the Merkle checksum tree. The hash value of the root node of the Merkle checksum tree can be used as the integrity check value for the entire video file. Each variable fragment, along with its corresponding fragment index, lightweight identifier, and the verification path from the leaf node to the root node, is encapsulated into a transmission data packet. The verification path includes the hash values ​​of all parent nodes traversed from the leaf node to the root node, used to verify the fragment's integrity at the receiving end.

[0062] Transmission priority weights are calculated based on the proximity of fragment indices and task urgency. Fragments with adjacent fragment indices are likely to have higher relevance and therefore can be assigned higher priority. Task urgency can be set according to the needs of the application scenario; for example, in real-time video transmission, newer fragments may have higher urgency. The fragment transmission sequence is determined based on the priority weights, while network latency is monitored and the scheduling strategy is adjusted in real time to adapt to network changes.

[0063] A point-to-point transmission channel is established using the WebRTC protocol to ensure real-time transmission and low latency. Transmission data packets are sent in a fragmented sequence to ensure ordered transmission of fragments. During transmission, the candidate addresses for Interactive Connectivity Establishment (ICE) are continuously updated to adapt to changes in network topology and improve transmission robustness.

[0064] In this embodiment, the receiver can verify the integrity of each fragment using a Merkle check tree, ensuring that data is not tampered with or lost during transmission. Calculating transmission priority weights based on fragment index proximity and task urgency to determine the fragment transmission sequence optimizes transmission order and improves efficiency. Real-time monitoring of network latency and adjustment of scheduling strategies adapt to network changes and reduce transmission delays. Establishing a point-to-point transmission channel via the WebRTC protocol and utilizing continuous updates to ICE candidate addresses improves transmission robustness and adapts to changes in network topology.

[0065] In some embodiments, in step S103 above, the step of extracting the visual feature vector of the first video data through the ResNet50 deep neural network, identifying the main object in the scene by combining it with the YOLOv8-tiny object detection model, and linking it with the CLIP image classification model to perform three-level classification of scene type and action category, specifically includes: The sampling interval is determined based on the dynamic relationship between the total video duration and the original frame rate, and the sampled frames are uniformly scaled to the preset pixel resolution through bilinear interpolation. For each sampled frame, a ResNet50 deep neural network is used to extract the visual feature vector of the first video data, and real-time object detection is performed through the YOLOv8-tiny object detection model, outputting a set of detection results including the coordinates of the bounding box, confidence score, and class ID; Based on the CLIP image classification model, the scene is classified by fusing the depth features of the visual feature vector with the HSV color histogram features to obtain the scene classification result. Based on the CLIP image classification model, the detection result set is classified into main objects. The action category is determined by continuous frame features and action descriptions to obtain the main object classification result and the action classification result.

[0066] In this embodiment, accurately extracting visual features, identifying the main objects in the scene, and classifying scene types and action categories are crucial in video processing and analysis. Traditional video analysis methods often suffer from inaccurate feature extraction, low object detection efficiency, and inaccurate scene and action classification. To address these issues, this invention provides a method that extracts visual feature vectors from video data using a ResNet50 deep neural network, identifies the main objects in the scene using a YOLOv8-tiny object detection model, and links it with a CLIP image classification model to perform a three-level classification of scene types and action categories, aiming to improve the accuracy and efficiency of video analysis.

[0067] Specifically, the total duration of the first video data is T seconds, and the original frame rate is F frames / second. Based on the dynamic relationship between the total video duration and the original frame rate, the sampling interval is determined to be Δt = T / N (where N is the preset number of sampling frames). N frames are extracted from the video according to the sampling interval as sampling frames. Each sampling frame is uniformly scaled to a preset pixel resolution (e.g., 224x224 pixels) using bilinear interpolation.

[0068] For each sampled frame, a ResNet50 deep neural network is used to extract visual feature vectors. The ResNet50 network consists of multiple convolutional layers, pooling layers, and fully connected layers, which can effectively extract deep features of the image. The sampled frame is input into the ResNet50 network, and the corresponding visual feature vector is output.

[0069] Real-time object detection is performed on each sampled frame using the YOLOv8-tiny object detection model. YOLOv8-tiny is a lightweight object detection model with fast detection speed and high detection accuracy. The model outputs a set of detection results, including the coordinates of the bounding box, confidence score, and class ID, which is used to identify the main object in the image.

[0070] Based on the CLIP image classification model, this method integrates deep features from visual feature vectors with HSV color histogram features to classify scenes. The CLIP model is a multimodal learning model capable of combining image and text information for classification. By fusing deep features and HSV color histogram features, the accuracy of scene classification is improved. Scene classification results are obtained, such as indoor, outdoor, urban, and natural.

[0071] Based on the CLIP image classification model, subject classification is performed on the detection result set. Action category determination is performed by analyzing continuous frame features and action descriptions. Continuous frame features include changes in the position and shape of the subject within consecutive frames. Action descriptions are textual descriptions of the subject's actions, such as "running" or "jumping." Subject classification results (e.g., people, animals, vehicles) and action classification results (e.g., walking, running, jumping) are obtained.

[0072] In this embodiment, visual feature vectors are extracted using a ResNet50 deep neural network, which accurately captures deep features of the image and improves the accuracy of feature extraction. The YOLOv8-tiny object detection model is employed, offering fast detection speed and high accuracy, meeting the requirements of real-time object detection. By fusing deep features of visual feature vectors with HSV color histogram features and combining them with the CLIP image classification model, the accuracy of scene classification is improved. Furthermore, by analyzing continuous frame features and action descriptions, combined with the CLIP image classification model, the action category of the main object is determined, enriching the content of video analysis.

[0073] In some embodiments, step S104 above, which involves constructing a video segment association map based on the second video data, specifically includes: The overall similarity between video segments in the second video data is calculated using a weighted fusion method. Based on the second video data, an undirected connection edge is established between two video segments whose comprehensive similarity exceeds a preset similarity threshold. The time interval relationship between video segments is detected, and a directed temporal edge is added between two video segments that meet the temporal proximity condition. Based on multiple video segments from the second video data and their undirected connections and right-directed temporal edges, a video segment association graph is constructed.

[0074] In this embodiment, understanding the relationships between video segments is crucial for tasks such as video content understanding, event detection, and video summarization in video processing and analysis. Traditional video segment association methods often only consider the content similarity between segments, ignoring the temporal relationships between them, resulting in inaccurate association results.

[0075] Specifically, a weighted fusion method is used to calculate the overall similarity between video segments, considering multiple similarity metrics such as color histogram similarity, motion feature similarity, and audio feature similarity. Based on the video data, undirected edges are established between two video segments whose overall similarity exceeds a preset similarity threshold, indicating that they have high content similarity. The time interval relationship between video segments is detected. For two video segments with a time interval less than a preset time threshold, they are considered to meet the temporal proximity condition, and directed temporal edges are added to them to represent their temporal relationship. Based on multiple video segments and their undirected edges and directed temporal edges, a video segment association graph is constructed to represent the complex relationships between video segments.

[0076] For example, the second video data is divided into multiple video segments, each with a fixed duration (e.g., 5 seconds) or dynamically divided based on changes in video content. For each pair of video segments, their color histogram similarity, motion feature similarity, and audio feature similarity are calculated. A weighted fusion method is used to combine these three similarity metrics to obtain the comprehensive similarity of each pair of video segments. The weighting coefficients can be adjusted according to specific application scenarios and needs. A preset similarity threshold (e.g., 0.8) is set. For two video segments whose comprehensive similarity exceeds the preset similarity threshold, an undirected connection edge is established, indicating that they have high content similarity. The time interval relationship between each pair of video segments is detected. A preset time threshold (e.g., 10 seconds) is set. For two video segments whose time interval is less than the preset time threshold, they are considered to meet the temporal proximity condition, and a directed temporal edge is added to them. The direction of the temporal edge indicates the temporal order of the video segments, pointing from the preceding segment to the following segment. Based on the multiple video segments of the video file and their undirected connection edges and directed temporal edges, a video segment association graph is constructed. In the graph, each node represents a video segment, undirected edges represent content similarity, and directed edges represent temporal relationships. By analyzing the structure and features of the graph, we can gain a deeper understanding of the relationships between video segments, providing strong support for subsequent tasks such as video content understanding, event detection, and video summarization.

[0077] In this embodiment, a weighted fusion method is used to calculate the comprehensive similarity, considering multiple similarity metrics and improving the accuracy of video segment association. By adding directed temporal edges, the temporal relationships between video segments are incorporated, making the association graph more consistent with the actual development of the video content. The constructed video segment association graph intuitively displays the relationships between video segments, enhancing readability and facilitating subsequent analysis and processing. The constructed video segment association graph can be applied to various tasks such as video content understanding, event detection, and video summarization generation, and has broad application prospects.

[0078] In some embodiments, step S104 above, which involves identifying the frame positions of each video segment, caching keyframes, and aligning multiple video segments in the second video data based on the video segment association map, specifically includes: Based on the video segment association map, the GOP analysis model is used to identify the frame positions of each video segment in the second video data to obtain the third video data; The key frames in the third video data are identified, and the key frames are preferentially buffered using a double-buffered queue to obtain the fourth video data. The frame rate difference between adjacent video segments of the fourth video data is calculated using an optical flow motion compensation algorithm, and the timestamps are dynamically adjusted to perform temporal alignment of multiple video segments.

[0079] In this embodiment, accurately identifying the frame positions of video segments, caching keyframes, and performing temporal alignment of multiple video segments are crucial during video processing. Traditional video processing methods often suffer from problems such as inaccurate frame position identification, inefficient keyframe caching, and difficulties in temporal alignment. To address these issues, this invention achieves efficient processing and alignment of video data through a GOP analysis model, a double-buffered queue, and an optical flow motion compensation algorithm.

[0080] Specifically, based on the video segment association map, the GOP (Group of Pictures) analysis model is used to identify the frame positions of each video segment in the video data. The GOP analysis model determines the start and end frames of each video segment by analyzing the encoding type and temporal relationship of the video frames, thus obtaining the processed video data. Keyframes are then identified in the processed video data. Keyframes are frames in a video segment that contain representative or important information, selected using specific algorithms or rules. Keyframes are preferentially buffered using a double-buffered queue. A double-buffered queue is a data structure used to store and manage keyframes in memory, ensuring that keyframes are processed and used preferentially to obtain video data for further processing. The optical flow motion compensation algorithm is used to calculate the frame rate differences between adjacent video segments in the further processed video data. The optical flow motion compensation algorithm analyzes the motion information between video frames to calculate the frame rate differences between adjacent video segments, providing a basis for temporal alignment. Based on the frame rate differences, timestamps are dynamically adjusted to achieve temporal alignment of multiple video segments. By adjusting the timestamps, multiple video segments are made consistent in time, achieving temporal alignment.

[0081] In this embodiment, the GOP analysis model accurately identifies the frame positions of video segments, improving the accuracy of frame position identification. Prioritizing keyframe caching using a double-buffered queue improves caching efficiency, ensuring keyframes are prioritized for subsequent processing and use. Utilizing an optical flow motion compensation algorithm, the frame rate difference between adjacent video segments can be accurately calculated, and timestamps are dynamically adjusted based on the difference, achieving temporal alignment of multiple video segments. The method of this invention combines the GOP analysis model, double-buffered queue, and optical flow motion compensation algorithm to achieve efficient processing and alignment of video data, improving the efficiency of video processing.

[0082] In some embodiments, step S105 above, which involves using a pyramid fusion algorithm to eliminate seam traces in the merged video streams and using a VMAF evaluation model to monitor video quality indicators in real time and dynamically adjust the quantization parameters of the merged video streams, specifically includes: The number of Laplacian pyramid layers is automatically calculated based on the video frame resolution of the merged video stream, and a multi-scale pyramid sequence is generated through Gaussian kernel convolution and downsampling operations. The pixel difference between adjacent video segments is calculated at each pyramid level of the multi-scale pyramid sequence, and a spatially varied fusion weight map is generated by using the Sigmoid function. Based on a dynamic weighted combination of deep convolution feature similarity, wavelet distortion metric, and temporal motion score, the VMAF evaluation model is used to monitor the quality of fused video in real time. Based on the output score of the VMAF evaluation model, the quantization parameters of the merged video stream are dynamically adjusted using a dynamic bitrate control algorithm.

[0083] In this embodiment, merging multiple video streams into a single continuous video stream is a common task in video processing. However, seams often appear during the merging process, affecting the visual quality of the video. Simultaneously, the quality of the merged video needs to be monitored and adjusted in real time to ensure the best user experience. To address these issues, this invention employs a pyramid fusion algorithm to eliminate seams in the merged video streams and uses a VMAF (Video Multimethod Assessment Fusion) evaluation model to monitor video quality metrics in real time and dynamically adjust the quantization parameters of the merged video streams. This method aims to improve the visual effect and overall quality of video merging.

[0084] Specifically, based on the video frame resolution of the merged video stream (assumed to be 1920x1080), the number of Laplacian pyramid layers (e.g., 4 layers) is automatically calculated. A multi-scale pyramid sequence is generated through Gaussian kernel convolution and downsampling operations. Each pyramid level contains video frames of different scales. At each pyramid level of the multi-scale pyramid sequence, the pixel difference between adjacent video segments is calculated. A spatially varying fusion weight map is generated using the Sigmoid function. This weight map is used to smoothly transition the pixel values ​​of adjacent video segments during the merging process, thereby eliminating seam artifacts. During the merging process, the depthwise convolutional features, wavelet distortion metric, and temporal motion score of the merged video frames are extracted in real time. These metrics are input into the VMAF evaluation model for dynamic weighted combination to obtain the quality score of the merged video. The VMAF evaluation model can comprehensively consider multiple quality metrics of the video, providing accurate video quality feedback. Based on the output score of the VMAF evaluation model, it is determined whether the quality of the merged video meets expectations. If the quality score is low, a dynamic bitrate control algorithm is used to dynamically adjust the quantization parameters of the merged video stream. By adjusting quantization parameters (such as reducing the quantization step size), the compression quality of the video can be improved, thereby optimizing the overall effect of the fused video.

[0085] In this embodiment, the pyramid fusion algorithm smoothly transitions pixel values ​​between adjacent video segments, effectively eliminating seams in the merged video streams. The VMAF evaluation model comprehensively considers multiple video quality metrics, providing accurate video quality feedback and facilitating timely problem identification and resolution. Through a dynamic bitrate control algorithm, the quantization parameters of the merged video streams can be dynamically adjusted based on the output score of the VMAF evaluation model, optimizing video quality. This invention eliminates seams, monitors and optimizes video quality in real time, thereby improving the user's viewing experience.

[0086] Reference Figure 2 An embodiment of the present invention provides an intelligent video merging processing system 2, the system 2 specifically comprising: The first processing module 201 is used to obtain the video file to be processed, and dynamically divide the video file to be processed into multiple variable segments using the Rabin fingerprinting block algorithm, and generate a unique hash value for each variable segment. The second processing module 202 is used to construct a Merkle check tree based on the unique hash value of all variable fragments and transmit the fragments via the WebRTC protocol; The third processing module 203 is used to acquire the first video data after the segmented transmission, extract the visual feature vector of the first video data through the ResNet50 deep neural network, identify the main object in the picture by combining the YOLOv8-tiny object detection model, and link the CLIP image classification model to perform three-level classification of scene type and action category to generate the second video data. The fourth processing module 204 is used to construct a video segment association map based on the second video data, and based on the video segment association map, to perform frame position identification, key frame caching, and temporal alignment of multiple video segments on the second video data to form a merged video stream. The fifth processing module 205 is used to eliminate seam traces in the merged video streams using a pyramid fusion algorithm, and to monitor video quality indicators in real time using a VMAF evaluation model, dynamically adjusting the quantization parameters of the merged video streams until the merged video streams meet the output standards.

[0087] It is understandable that, such as Figure 1 The content of the intelligent video merging processing method embodiments shown is applicable to the intelligent video merging processing system embodiments of this invention. The specific functions implemented by the intelligent video merging processing system embodiments of this invention are the same as those shown in the examples. Figure 1 The intelligent video merging processing method shown in the embodiment is the same, and the beneficial effects achieved are the same as those described above. Figure 1 The beneficial effects achieved by the intelligent video merging processing method embodiment shown are also the same.

[0088] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0089] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0090] Reference Figure 3 The present invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, it implements the intelligent processing method for video merging as described in any of the above methods.

[0091] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0092] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0093] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0094] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an intelligent video merging processing method as described in any of the above methods.

[0095] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0096] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0098] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. An intelligent video merging processing method, characterized in that, The method specifically includes: The video file to be processed is obtained, and the Rabin fingerprinting algorithm is used to dynamically divide the video file into multiple variable segments, generating a unique hash value for each variable segment. A Merkle check tree is constructed based on the unique hash value of all variable fragments, and fragmented transmission is performed via the WebRTC protocol; The first video data after being fragmented and transmitted is obtained. The visual feature vector of the first video data is extracted by the ResNet50 deep neural network. The YOLOv8-tiny object detection model is combined to identify the main object in the picture. The CLIP image classification model is linked to perform three-level classification of scene type and action category to generate the second video data. Based on the second video data, a video segment association graph is constructed. Based on the video segment association graph, the frame position of each video segment is identified, key frame is cached, and multiple video segments are time-series aligned to form a merged video stream. The pyramid fusion algorithm is used to eliminate the seams in the merged video streams, and the VMAF evaluation model is used to monitor video quality indicators in real time and dynamically adjust the quantization parameters of the merged video streams until the merged video streams meet the output standards.

2. The method according to claim 1, characterized in that, The Rabin fingerprinting algorithm is used to dynamically divide the video file to be processed into multiple variable segments, and a unique hash value is generated for each variable segment. Specifically, this includes: The video file to be processed is scanned byte by byte through a sliding window, and the fingerprint value of each sliding window is calculated using the Rabin fingerprint algorithm. When the fingerprint value of any sliding window is greater than the preset fingerprint threshold n times consecutively, a temporary fragmentation boundary marker is triggered, and the start code of the first I-frame after the temporary fragmentation boundary marker appears is set as the final fragmentation cutting point. The video file to be processed is divided into multiple variable segments based on the average compression efficiency by dynamically selecting the segment size threshold and the final segment cutting pad and segment size threshold. Calculate the full SHA3-256 hash value of each variable segment and use the full SHA3-256 hash value as the unique hash value of the variable segment.

3. The method according to claim 1, characterized in that, The process of constructing a Merkle check tree based on the unique hash values ​​of all variable fragments and transmitting the fragments via the WebRTC protocol specifically includes: Use the first k bits of the unique hash value of each variable partition as a lightweight identifier, and set the leaf nodes according to the lightweight identifier; Based on the leaf nodes, the parent node is formed by recursively calculating the connection digest of the hash values ​​of adjacent nodes until a complete binary tree structure is generated, thus obtaining the Merkle check tree. Each variable fragment, along with its corresponding fragment index, lightweight identifier value, and verification path from the leaf node to the root node, is encapsulated into a transmission data packet. The transmission priority weight is calculated based on the proximity and urgency of the fragment index to determine the fragment transmission sequence, while network latency is monitored and the scheduling strategy is adjusted in real time. A point-to-point transmission channel is established using the WebRTC protocol, and data packets are sent according to the fragmented transmission sequence, while the ICE candidate address is continuously updated during transmission.

4. The method according to claim 1, characterized in that, The process involves extracting visual feature vectors from the first video data using a ResNet50 deep neural network, combining this with a YOLOv8-tiny object detection model to identify the main objects in the scene, and linking this with a CLIP image classification model to perform three-level classification of scene type and action category. Specifically, this includes: The sampling interval is determined based on the dynamic relationship between the total video duration and the original frame rate, and the sampled frames are uniformly scaled to the preset pixel resolution through bilinear interpolation. For each sampled frame, a ResNet50 deep neural network is used to extract the visual feature vector of the first video data, and real-time object detection is performed through the YOLOv8-tiny object detection model, outputting a set of detection results including the coordinates of the bounding box, confidence score, and class ID; Based on the CLIP image classification model, the scene is classified by fusing the depth features of the visual feature vector with the HSV color histogram features to obtain the scene classification result. Based on the CLIP image classification model, the detection result set is classified into main objects. The action category is determined by continuous frame features and action descriptions to obtain the main object classification result and the action classification result.

5. The method according to claim 1, characterized in that, The construction of the video segment association map based on the second video data specifically includes: The overall similarity between video segments in the second video data is calculated using a weighted fusion method. Based on the second video data, an undirected connection edge is established between two video segments whose comprehensive similarity exceeds a preset similarity threshold. The time interval relationship between video segments is detected, and a directed temporal edge is added between two video segments that meet the temporal proximity condition. Based on multiple video segments from the second video data and their undirected connections and right-directed temporal edges, a video segment association graph is constructed.

6. The method according to claim 1, characterized in that, The process of identifying frame positions for each video segment, caching keyframes, and performing temporal alignment of multiple video segments based on the video segment association graph specifically includes: Based on the video segment association map, the GOP analysis model is used to identify the frame positions of each video segment in the second video data to obtain the third video data; The key frames in the third video data are identified, and the key frames are preferentially buffered using a double-buffered queue to obtain the fourth video data. The frame rate difference between adjacent video segments of the fourth video data is calculated using an optical flow motion compensation algorithm, and the timestamps are dynamically adjusted to perform temporal alignment of multiple video segments.

7. The method according to claim 1, characterized in that, The process involves employing a pyramid fusion algorithm to eliminate seams in the merged video streams and using a VMAF evaluation model to monitor video quality metrics in real time and dynamically adjust the quantization parameters of the merged video streams. Specifically, this includes: The number of Laplacian pyramid layers is automatically calculated based on the video frame resolution of the merged video stream, and a multi-scale pyramid sequence is generated through Gaussian kernel convolution and downsampling operations. The pixel difference between adjacent video segments is calculated at each pyramid level of the multi-scale pyramid sequence, and a spatially varied fusion weight map is generated by using the Sigmoid function. Based on a dynamic weighted combination of deep convolution feature similarity, wavelet distortion metric, and temporal motion score, the VMAF evaluation model is used to monitor the quality of fused video in real time. Based on the output score of the VMAF evaluation model, the quantization parameters of the merged video stream are dynamically adjusted using a dynamic bitrate control algorithm.

8. An intelligent video merging processing system, characterized in that, The system specifically includes: The first processing module is used to obtain the video file to be processed, and uses the Rabin fingerprinting algorithm to dynamically divide the video file to be processed into multiple variable segments, and generates a unique hash value for each variable segment. The second processing module is used to construct a Merkle check tree based on the unique hash value of all variable fragments and transmit the fragments via the WebRTC protocol; The third processing module is used to acquire the first video data after the segmented transmission, extract the visual feature vector of the first video data through the ResNet50 deep neural network, identify the main object in the picture by combining the YOLOv8-tiny object detection model, and link the CLIP image classification model to perform three-level classification of scene type and action category to generate the second video data. The fourth processing module is used to construct a video segment association map based on the second video data. Based on the video segment association map, it performs frame position identification, key frame caching, and temporal alignment of multiple video segments on the second video data to form a merged video stream. The fifth processing module is used to eliminate seam traces in the merged video streams using the pyramid fusion algorithm, and to monitor video quality indicators in real time using the VMAF evaluation model, dynamically adjusting the quantization parameters of the merged video streams until the merged video streams meet the output standards.

9. A computer device, characterized in that, include: The memory and processor, and the computer program stored in the memory, which, when executed on the processor, implement the intelligent processing method for video merging as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the intelligent processing method for video merging as described in any one of claims 1 to 7.

Images