Multi-terminal video collaboration method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a multi-terminal video collaboration method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: acquiring continuous image frames, and acquiring a main code stream and a bypass frame according to the continuous image frames; outputting a main code stream and acquiring a network bandwidth condition; determining a transmission strategy for the bypass frame based on the network bandwidth condition, and packaging the transmission strategy into a data packet for transmission; the transmission policy includes determining a proportion of bypass frames. According to the invention, the multi-terminal cooperative processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal transmission technology, and in particular to a multi-terminal video collaboration method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] In multi-terminal collaboration scenarios, such as capability sharing between a camera and a TV, interactive home services like motion-sensing games and AR sports can be supported. Current common collaboration models typically rely on the camera pushing H.265 / H.264 encoded video streams. The TV decodes these streams, writes the decoded RAW image data to shared memory, and then reads it through the virtual video driver (VD) in the HAL layer for use by upper-layer applications. However, existing multi-terminal video collaboration methods suffer from low collaborative processing efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a multi-terminal video collaboration method, apparatus, computer equipment, readable storage medium, and program product that can improve the efficiency of collaborative processing in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a multi-terminal video collaboration method, applied at the acquisition end, including:

[0005] Acquire consecutive image frames, and obtain the main bitstream and bypass frames based on the consecutive image frames;

[0006] Output the main bitstream and obtain network bandwidth status;

[0007] Based on the network bandwidth conditions, a transmission strategy is determined for the bypass frames, and they are encapsulated into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

[0008] In one embodiment, determining the transmission strategy for the bypass frame based on the network bandwidth conditions and encapsulating it into a data packet for transmission includes:

[0009] Whether to trigger the transmission of the bypass frame is determined based on the available bandwidth.

[0010] The allocable bandwidth of the bypass frame is determined based on the target bitrate of the main bitstream, and the frequency of the bypass frame is calculated based on the allocable bandwidth and the single frame size of the bypass frame.

[0011] The bypass frames are encapsulated into data packets based on their frequency and then transmitted.

[0012] In one embodiment, the method further includes:

[0013] If the main stream and the bypass frame share the same transmission channel for transmission, the main stream and the bypass frame are distinguished based on the identification field.

[0014] In one embodiment, the step of acquiring consecutive image frames and acquiring the main bitstream and bypass frames based on the consecutive image frames includes:

[0015] The continuous image frames are acquired by the image acquisition module of the acquisition terminal;

[0016] The consecutive image frames are preprocessed and color compressed;

[0017] The video encoder at the acquisition end generates a standard main bitstream based on the continuous image frames, and selects some key frames and / or periodic frames of the continuous image frames to form the bypass frames.

[0018] Secondly, this application also provides a multi-terminal video collaboration method applied to the decoding end, the method comprising:

[0019] Receive data packets and determine the frame type of the data packets, the frame type including main stream and bypass frames;

[0020] The data packet is parsed based on the frame type to obtain video frames written to different memory regions; the memory regions include at least those that do not pass through the video decoder.

[0021] The corresponding video frames are obtained from the different memory regions based on the frame scheduling strategy.

[0022] In one embodiment, parsing the data packet based on the frame type to obtain video frames written to different memory regions includes:

[0023] If the frame type of the data packet is a main bitstream, the data packet is parsed based on the video decoder to generate video frames and write them to shared memory;

[0024] If the frame type of the data packet is a bypass frame, the data packet is parsed based on the bypass parser to generate a video frame and write it to the high-order memory.

[0025] In one embodiment, obtaining the corresponding video frame from the different memory regions based on the frame scheduling strategy includes:

[0026] Instruct the memory manager to record the frame information of the data packets written to the high-order memory;

[0027] The virtual driver, based on the frame information, schedules the frame data of the corresponding video frame from the shared memory or the high-order memory for the hardware abstraction layer to read.

[0028] Thirdly, this application also provides a multi-terminal video collaboration device, applied at the acquisition end, including:

[0029] The acquisition unit is used to acquire continuous image frames and acquire the main bitstream and bypass frames based on the continuous image frames;

[0030] A bandwidth sensing unit is used to output the main bitstream and obtain network bandwidth status.

[0031] An encoding and transmission unit is used to determine a transmission strategy for the bypass frames based on the network bandwidth conditions, and encapsulate them into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

[0032] Fourthly, this application also provides a multi-terminal video collaboration device applied at the decoding end, comprising:

[0033] A receiving unit is configured to receive data packets and determine the frame type of the data packets, wherein the frame type includes the main stream and bypass frames;

[0034] The parsing unit is used to parse the data packet based on the frame type to obtain video frames written to different memory regions; the memory regions include at least the memory regions that do not go through the video decoder;

[0035] The video reading unit is used to obtain the corresponding video frames from the different memory regions based on the frame scheduling strategy.

[0036] Fifthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods described in the first and second aspects.

[0037] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the first and second aspects.

[0038] In a seventh aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the first and second aspects.

[0039] The aforementioned multi-terminal video collaboration method, apparatus, computer equipment, readable storage medium, and program product acquire continuous image frames at the acquisition end, and acquire the main bitstream and bypass frames based on the continuous image frames; output the main bitstream and acquire network bandwidth status; determine a transmission strategy for the bypass frames based on the network bandwidth status, and encapsulate them into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames, receiving data packets at the decoding end and determining the frame type of the data packets, the frame type including the main bitstream and bypass frames; parsing the data packets based on the frame type to acquire video frames written to different memory regions; the memory regions include at least memory regions that do not pass through the video decoder; and acquiring the corresponding video frames from the different memory regions based on the frame scheduling strategy, thereby selectively transmitting based on network bandwidth status and allowing bypass frames to bypass resource-intensive video collaboration steps, thus improving collaborative processing efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a multi-terminal video collaboration method applied to the acquisition end in one embodiment;

[0042] Figure 2 This is a flowchart illustrating a multi-terminal video collaboration method applied to the decoding end in one embodiment;

[0043] Figure 3 This is a schematic diagram of the overall process of a multi-terminal video collaboration method in one embodiment;

[0044] Figure 4 This is a system architecture diagram of a multi-terminal video collaboration method applied in one embodiment;

[0045] Figure 5 This is a specific implementation flow of a multi-terminal video collaboration method in one embodiment;

[0046] Figure 6 This is a structural block diagram of a multi-terminal video collaboration device applied to the acquisition end in one embodiment;

[0047] Figure 7 This is a structural block diagram of a multi-terminal video collaboration device applied to the decoding end in one embodiment;

[0048] Figure 8This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] In multi-terminal collaboration scenarios, such as capability sharing between a camera and a TV, interactive home services like motion-sensing games and AR sports can be supported. Current common collaboration models typically rely on the camera pushing H.265 / H.264 encoded video streams. The TV decodes these streams, writes the decoded RAW image data to shared memory, and then reads it through the virtual video driver (VD) of the HAL layer for use by upper-layer applications.

[0052] However, this mode involves frequent reading and writing of large amounts of image data, which puts a lot of pressure on the set-top box's decoding capabilities, memory bandwidth, and CPU performance. This can easily lead to problems such as device overheating and system load spikes, resulting in a decline in the user experience, such as screen stuttering, screen flickering, or response delays.

[0053] For example, in existing multi-terminal video collaboration solutions, decoded frame data on the TV end is mostly stored in low-end memory. The HAL layer or virtual driver needs to copy the data multiple times to access it, leading to increased memory bandwidth and CPU load in high-resolution scenarios, easily causing latency and performance bottlenecks. Frame transmission lacks dynamic adjustment based on terminal performance and bandwidth; bypass frames often require manual configuration and cannot be automatically enabled or have their FPS adjusted according to the operating status, resulting in wasted bandwidth or stuttering. Most solutions rely on cloud forwarding, lacking optimization for direct LAN connections, making it difficult to leverage the high-speed forwarding capabilities of home gateways, resulting in link redundancy and high latency.

[0054] More specifically, existing multi-terminal collaboration solutions for sharing video capabilities have significant performance bottlenecks in actual deployments, mainly in the following aspects:

[0055] Frequent shared memory read / write operations put significant strain on system performance: Video streams (such as H.264 / H.265) uploaded from cameras need to be decoded by the set-top box, and the decoded frames are then written to shared memory for use by upper-layer applications. Due to the large size of image frames and frequent data read / write operations to shared memory, high memory bandwidth usage and frequent bus conflicts are easily generated, increasing the overall load on the SoC.

[0056] The multi-level decoding link is complex and has high processing latency: Each frame of image data after decoding needs to be transmitted to the upper layer application through the HAL layer, multi-level cache and virtual device driver, resulting in a long data transmission link and large frame latency jitter, which makes it difficult to meet the real-time requirements of business scenarios such as motion interaction and AR motion.

[0057] Based on the aforementioned traditional technologies, this application provides a multi-terminal video collaboration method, which can also be a bandwidth-aware intelligent bypass decoding method for video streams. When bandwidth is sufficient, the camera can transmit standard video streams and MJPEG image data in parallel. After receiving the MJPEG, the television receives the frame data directly from the HAL layer, bypassing the decoder and shared memory process, effectively reducing resource consumption and improving collaborative processing efficiency.

[0058] Specifically, to address the issues of high memory read / write overhead, lengthy processing links, and high system power consumption in multi-terminal video collaboration, this application proposes an intelligent video stream decoding and frame access method for real-time multi-terminal collaboration. When the camera detects sufficient bandwidth (e.g., through RTCP evaluation), it synchronously sends some YUV or MJPEG image frames in addition to the encoded standard video stream (H.264 / H.265). The television receives these frames and directly writes them into high-order memory, which is then used by the HAL layer for application calls (e.g., motion-sensing games). This method skips the traditional process of writing decoded frames into shared memory, allowing the virtual device driver (VD) to fuse and read shared and bypass frames, achieving low latency and efficient access.

[0059] Furthermore, the intelligent video stream decoding and frame access method for multi-terminal real-time collaboration proposed in this application is applicable to the efficient collaboration of smart cameras and set-top boxes / TV terminals in scenarios such as home entertainment, AR interaction, and motion sensing applications. This method introduces bandwidth-aware and bypass frame transmission mechanisms without altering the standard video encoding and transmission mechanism, thereby improving the access efficiency and processing performance of video frames between multiple terminals.

[0060] In the method proposed in this application embodiment, when the camera detects sufficient network bandwidth (e.g., through RTCP or link quality feedback), in addition to the regular H.264 / H.265 encoded video stream, it synchronously sends a portion of YUV / MJPEG image frames according to a dynamic strategy.

[0061] During the reception process on the TV, the network card driver determines whether the received data is a bypass frame. If so, the bypass frame data is written directly to the high-order memory, and the virtual driver is notified to read it directly. The HAL layer provides access for applications (such as motion-sensing games), skipping the traditional processing link of writing to shared memory after decoding.

[0062] Meanwhile, the Virtual Device Driver (VD) enables automatic fusion and scheduling of shared and bypass frames, ensuring the continuity of upper-layer services and data consistency. This method effectively reduces the decoding load and memory bandwidth pressure on the terminal SoC, reduces power consumption and thermal load, and improves video response time and service stability.

[0063] It is understandable that, compared to traditional single-stream video decoding schemes, the method proposed in this application combines video decoding with bypass frame pass-through. More specifically, in terms of camera output, this application combines H.265 / H.264 encoded streams with bypass YUV / MJPEG output; in terms of frame decoding, this application skips decoding of some frames and directly uses bypass frames; in terms of frame writing path, this application writes video frames to low-order memory after decoding and then copies them to shared memory, while bypass frames are directly written to high-order memory without intermediate copying; in terms of VD memory access, this application integrates access from shared memory and high-order memory; in terms of bandwidth adaptation mechanism, this application uses dynamic bypass ratio adjustment; therefore, based on the above methods, compared to traditional schemes with high CPU load, large bandwidth consumption, and easy overheating due to high decoding / memory load, this application reduces decoding load and memory bandwidth pressure, lowers SoC load, and alleviates heat dissipation pressure.

[0064] Furthermore, in terms of memory architecture, this application's embodiments reduce multiple copy operations during the transmission of video to the HAL layer after decoding by optimizing high-low memory mapping and shared memory access paths; in terms of frame scheduling, a bandwidth awareness and terminal performance monitoring mechanism is introduced to achieve automatic activation of bypass frames and dynamic adjustment of FPS; in terms of system architecture, for collaborative scenarios where the camera and TV are on the same home LAN, a low-latency, bandwidth-adaptive direct connection transmission process is constructed, significantly improving the real-time performance and stability of video collaboration, outperforming the performance of existing fixed configuration and cloud forwarding architectures.

[0065] This application further proposes an intelligent video decoding and bypass frame dynamic scheduling method for real-time multi-terminal collaboration in a local area network. Compared with existing multi-terminal collaboration schemes that suffer from fixed decoding paths, high memory copy loads, and a lack of adaptive frame scheduling, this method achieves high efficiency and dynamic resource optimization in the video processing link. The specific optimization methods are as follows:

[0066] Decoding path and memory access optimization: This solution writes bypass frames (YUV / MJPEG) directly into high-order memory and allows seamless access by the virtual driver and HAL layer, skipping the traditional multiple memory copy process, significantly reducing bus bandwidth usage and CPU load, and maintaining low-latency and stable output in high-resolution and high-frame-rate scenarios.

[0067] Bypass frame intelligent scheduling mechanism: Through the bandwidth perception and performance monitoring module at the camera end, the link and terminal performance status are judged in real time, and the bypass frame FPS and ratio are automatically enabled or adjusted to maximize the use of remaining bandwidth without affecting the main bitstream, effectively balancing image quality and latency.

[0068] The embodiments of this application complement each other in terms of decoding link optimization, bandwidth adaptation, and local area network terminal collaboration, which is superior to existing fixed path and manual configuration strategies. They are especially suitable for application scenarios with extremely high requirements for latency and multi-terminal synchronization, such as AR sports and motion-sensing games.

[0069] In practical applications, the solution of this application optimizes the decoding path and adaptive frame scheduling of the multi-terminal video collaboration process, which is suitable for video service scenarios with high-frequency interaction of multiple terminals in a local area network. It significantly reduces the memory bandwidth and CPU load of the decoding end, improves the smoothness of the picture and the response speed, and ensures the real-time performance and stability of multi-terminal collaboration.

[0070] This application embodiment can be deployed in multimedia terminals such as home cameras and televisions. Through mechanisms such as high and low memory hierarchical management, direct writing of bypass frames, and dynamic FPS adjustment, it ensures stable output of the main video stream while making full use of the remaining bandwidth to transmit high-quality bypass frames, reducing redundant copies and link latency, and achieving parallel optimization of video rendering and application calls.

[0071] Possible future applications of this application include: Home entertainment and interactive scenarios (smart home / IPTV): In scenarios such as motion-sensing games and AR sports, the camera can capture and transmit high-quality bypass frames to the TV application in real time, achieving low-latency motion capture and image rendering, and improving the interactive experience. Remote collaboration and education scenarios (SME ICT / smart office): In multi-party video conferencing or interactive classrooms, bypass frames can be directly supplied to specific applications (such as whiteboard recognition and motion tracking), improving the real-time performance and clarity of collaboration tools and avoiding the impact of main stream decoding delays on the experience.

[0072] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0073] Before introducing the specific embodiments of this application, the technical terms involved in this application will be explained:

[0074] Television terminal: Terminal devices that receive and play videos, including set-top boxes, smart TVs, projectors, etc.

[0075] Camera: WiFi camera (non-USB), which uses wireless / wired access to the home LAN. Its main function is security monitoring, and it can also be used as a peripheral device for TV in multi-device collaboration scenarios.

[0076] GOP (Group of Pictures): A structural unit in video coding consisting of a group of consecutive video frames, usually composed of one I-frame and several P-frames and B-frames, used to balance compression efficiency and random access capability.

[0077] I-frame (Intra-coded Frame): An intra-coded frame that contains complete image information, can be decoded independently, and is a key reference frame for the Group of Pictures (GOP).

[0078] P-frame (Predicted Frame): A predictive coded frame that only records the differences between the frame and the reference frame. It requires I-frames or other P-frames for decoding.

[0079] YUV: A color encoding format that includes luminance (Y) and chrominance (U, V) components, commonly used for representing video image data.

[0080] MJPEG (Motion JPEG): An encoding method that independently compresses each frame of a video into a JPEG image, featuring simple decoding and independent frame-to-frame processing.

[0081] RTP (Real-time Transport Protocol): A real-time transport protocol used to transmit audio and video data over IP networks, supporting real-time guarantee mechanisms such as timestamps and sequence numbers.

[0082] UDP (User Datagram Protocol): A connectionless, low-latency transport protocol commonly used for real-time audio and video transmission.

[0083] RTCP (Real-time Transport Control Protocol): A real-time transport control protocol used in conjunction with RTP to transmit control information such as quality feedback and bandwidth estimation.

[0084] HAL (Hardware Abstraction Layer): This layer abstracts away hardware differences and provides a unified hardware access interface for upper-layer applications.

[0085] VD (Virtual Driver): A virtual device driver used to virtualize device functionality within the operating system, enabling efficient data transfer and access between different memory regions and applications.

[0086] Shared memory: an inter-process communication mechanism that allows multiple processes to directly access the same physical memory area, used for efficient transmission of video frame data and reducing copying overhead.

[0087] High memory: The memory region located in the high address area of ​​the system, which can be used to store large data blocks such as bypass frames, reducing copy operations with shared memory.

[0088] Low memory: The memory region in the low address range of the system. It is usually the default output target of the video decoder and can only be accessed by other modules by copying.

[0089] Bypass Frame: An image frame that is compressed and transmitted directly by the encoder and used by the application without going through the regular video decoding path, such as a YUV or MJPEG frame.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] In one exemplary embodiment, such as Figure 1 As shown, a multi-terminal video collaboration method is provided. Taking the application of this method to the acquisition end as an example, the method includes the following steps 202 to 206. Wherein:

[0092] Step 202: Obtain continuous image frames, and obtain the main bitstream and bypass frames based on the continuous image frames.

[0093] For example, consecutive image frames are acquired, and the acquired consecutive image frames are divided into main stream and bypass frames according to preset rules. By dividing consecutive image frames into main stream and bypass frames, a basis is provided for performing different collaborative processing on consecutive image frames.

[0094] Step 204: Output the main bitstream and obtain network bandwidth status.

[0095] The network bandwidth status may include information that can be obtained through RTCP, such as bandwidth size, packet loss rate, and end-to-end latency (TTL, Time-To-Live). This application embodiment is not limited to this.

[0096] Specifically, the main bitstream is output and transmitted, and the network bandwidth status during the transmission of the main bitstream is obtained, thereby determining whether the network bandwidth is sufficient.

[0097] Step 206: Determine the transmission strategy for bypass frames based on network bandwidth conditions, and encapsulate them into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

[0098] Specifically, based on network bandwidth conditions, including whether the network bandwidth is sufficient, a transmission strategy for bypass frames is determined. The bypass frames are then encapsulated into data packets and transmitted simultaneously with the main stream according to the transmission strategy. The transmission strategy includes determining the proportion of bypass frames during transmission, thereby enabling real-time transmission and adjustment of bypass frames based on network bandwidth conditions, thus constructing a bandwidth-adaptive transmission.

[0099] In the aforementioned multi-terminal video collaboration method, continuous image frames are acquired, and the main bitstream and bypass frames are obtained based on these frames. The main bitstream is output, and the network bandwidth status is obtained. Based on the network bandwidth status, a transmission strategy for bypass frames is determined and encapsulated into data packets for transmission. The transmission strategy includes determining the proportion of bypass frames, thus proposing a dynamically adjustable bypass frame mechanism for multi-terminal collaborative video stream transmission and encoding / decoding. This allows the acquisition end to increase bypass frame transmission based on network conditions and dynamically adjust the number and size of bypass frames according to the network environment, significantly improving the real-time performance and stability of video collaboration.

[0100] Optionally, refer to Figure 3 Multi-terminal video collaboration methods applied to the acquisition end can include: acquisition, YUV conversion, video encoding, MJPEG conversion, and packet transmission.

[0101] In an exemplary embodiment, step 206, determining a transmission strategy for bypass frames based on network bandwidth conditions and encapsulating them into data packets for transmission, includes:

[0102] Whether to trigger the transmission of bypass frames is determined based on the available bandwidth.

[0103] The allocable bandwidth of the bypass frame is determined based on the target bitrate of the main bitstream, and the frequency of the bypass frame is calculated based on the allocable bandwidth and the single frame size of the bypass frame.

[0104] Bypass frames are encapsulated into data packets and transmitted based on their frequency.

[0105] The frequency of bypass frames can also refer to the number of bypass frames.

[0106] Specifically, a smart frame scheduling and transmission strategy is formed by determining whether to trigger the transmission of bypass frames based on the available bandwidth; determining the allocable bandwidth of bypass frames based on the target bit rate of the main bitstream; calculating the frequency of bypass frames based on the allocable bandwidth and the single frame size of bypass frames; and encapsulating bypass frames into data packets and transmitting them based on the frequency of bypass frames.

[0107] For example, taking a camera as the acquisition end and a TV as the decoding end, the intelligent frame scheduling and transmission strategy may include the following steps:

[0108] 1) During the continuous transmission of the main stream to the TV, the bandwidth awareness module assesses the network bandwidth status in real time using methods such as RTCP and UDP ACK. Figure 5 Network bandwidth assessment in the context of network bandwidth.

[0109] 2) The frame scheduling module dynamically determines whether to enable bypass frame transmission and its proportion based on evaluation results, service type, GOP structure, etc. Figure 5 The transmission decision that determines the number of bypass frames, and the specific calculation process and formula for the bypass frame transmission ratio are as follows:

[0110] Average I / P frame size ratio: α = S_I / S_P;

[0111] Total main stream data volume per second (estimated):

[0112] R_main ≈ (S_I + (G-1) × S_P) × F / G × 8 / 1000 Mbps;

[0113] Bypass MJPEG bitrate estimation:

[0114] R_BYP = S_BYP × F_BYP × 8 / 1000 Mbps;

[0115] Total bitrate constraints for bypass and main streams (based on bandwidth):

[0116] R_main + R_BYP ≤ (1-β) × R_avail;

[0117] The maximum value of F_BYP can be obtained using the above formula. The specific parameters involved in the formula are shown in the table below:

[0118] parameter symbol unit illustrate Video bitrate (total bandwidth) R_total Mbps Overall bitrate after encoding the main video stream GOP length G Frames A group consisting of an I-frame and several P / B frames. I frame size S_I KB Average size of a single I-frame P-frame size S_P KB Average size of a single P-frame I / P frame size ratio α — α = S_I / S_P, used to estimate frame distribution Frame rate F fps Frames per second, such as 30fps MJPEG / YUV bypass frame size S_BYP KB Average size of a single YUV / MJPEG frame after compression Bypass frame frequency f_BYP Frames per second How many YUV / MJPEG frames are inserted per second? Available bandwidth R_avail Mbps After RTCP detection, the camera side can use uplink bandwidth. Secure bandwidth ratio β — If the reserved ratio is 20%, then the effective available bandwidth is (1-β)·R_avail Bypass frame rate R_BYP Mbps The actual bitrate used for bypass frames = S_BYP × f_BYP × 8 / 1000 Main stream bitrate R_main Mbps Main stream bitrate for H.264 / H.265

[0119] In one exemplary embodiment, the method further includes:

[0120] If the main stream and bypass frames share the same transmission channel, the main stream and bypass frames are distinguished based on the identification field.

[0121] The main stream can refer to the video stream.

[0122] Specifically, based on the transmission strategy of video stream and bypass frame, if the main stream and bypass frame share the same transmission channel, the main stream and bypass frame are distinguished based on the identification field.

[0123] Exemplary reference Figure 5 The transmission strategies for video streams and bypass frames can include in-band transmission and out-of-band transmission. More specifically, refer to... Figure 5 In multi-terminal video collaboration systems, after bypass frame packing and video frame packing are performed, the camera side needs to simultaneously transmit the standard main stream (such as H.265 / H.264 encoded frames) and bypass frames (such as YUV and MJPEG compressed frames) to achieve efficient frame access capabilities and a better terminal collaboration experience. To achieve unified management and decoupled transmission of dual-channel data, the main stream and bypass frames can be transmitted in the following two ways:

[0124] Method 1, In-band transmission: The main stream and bypass frame data share the same transmission channel. When the data is encapsulated at the UDP layer, the data type is distinguished by a custom payload identifier field.

[0125] Method 2, out-of-band transmission: The main stream and bypass frames are transmitted separately using different ports or stream IDs, reducing the risk of frame out-of-order transmission. This method is suitable for scenarios with sufficient bandwidth resources and terminals with multi-channel parsing capabilities.

[0126] For the in-band transmission mode of Method 1, in order to distinguish different types of frames and maintain compatibility with the existing RTP encapsulation mechanism, a custom identifier field needs to be added to the UDP payload header. The specific identifier fields can be found in the table below:

[0127] Byte Offset Fields illustrate 0 Frame header Fixed value such as 0xAB 1 Frame type identifier 0x00 = Main stream RTP frame; 0x11 = YUV frame (bypass); 0x12 = MJPEG frame (bypass) 2~N Frame data topic payload RTP data or YUV / MJPEG

[0128] In an exemplary embodiment, step 202, obtaining consecutive image frames, and obtaining the main bitstream and bypass frames based on the consecutive image frames, includes:

[0129] Continuous image frames are acquired through the image acquisition module at the acquisition end;

[0130] Preprocessing and color compression of consecutive image frames;

[0131] The video encoder at the acquisition end generates a standard main bitstream based on continuous image frames, and selects some key frames and / or periodic frames from the continuous image frames to form bypass frames.

[0132] Among them, consecutive image frames can refer to video frames.

[0133] Specifically, continuous image frames are acquired through the image acquisition module of the acquisition end; the continuous image frames are preprocessed and color compressed; and a standard main bitstream is generated based on the continuous image frames through the video encoder of the acquisition end, and some key frames and / or periodic frames of the continuous image frames are selected to form bypass frames.

[0134] For example, the acquisition end can be a camera, which enables video frame acquisition and parallel encoding, as shown in the following example. Figure 5 First, the camera image acquisition module acquires continuous image frames (RAW / YUV). After image preprocessing and color compression, the image frames are simultaneously sent to two encoding paths. One encoding path generates a standard main bitstream (such as H.264 / H.265) by performing YUV subsampling through a video encoder. The other encoding path selects some key frames or periodic frames and encodes them using MJPEG or lightweight YUV to form bypass frames.

[0135] In one exemplary embodiment, such as Figure 2 As shown, a multi-terminal video collaboration method is provided. Taking the application of this method on the decoding end as an example, the method includes the following steps 302 to 306. Wherein:

[0136] Step 302: Receive data packets and determine the frame type of the data packets. The frame type includes main stream and bypass frames.

[0137] Specifically, by acquiring data from the acquisition end, receiving the main bitstream and bypass frame data packets, and determining the frame type based on the data information in the data packets, different processing methods can be provided according to different frame types.

[0138] Step 304: Parse the data packets based on the frame type to obtain the video frames written to different memory regions; the memory regions include at least those that do not pass through the video decoder.

[0139] Specifically, based on several different types of data of different frame types in the data packet, different data are sent to different memory areas corresponding to different processing paths, thereby performing different processing methods. The memory areas include at least the memory areas that do not go through the video decoder. That is to say, the video frames include at least the video frames that do not need to be processed by the video decoder, thereby reducing the decoding pressure on the video decoder at the decoding end and improving processing performance.

[0140] Step 306: Obtain the corresponding video frames from different memory regions based on the frame scheduling strategy.

[0141] Specifically, the decoding end obtains corresponding video frames from different memory regions based on a frame scheduling strategy, thereby achieving efficient scheduling, reading, and presentation of video frames.

[0142] In the aforementioned multi-terminal video collaboration method, data packets are received and their frame types are determined. Frame types include main stream and bypass frames. Data packets are parsed based on frame types to obtain video frames to be written to different memory regions. Each memory region includes at least those that do not pass through a video decoder. The corresponding video frames are then retrieved from different memory regions based on a frame scheduling strategy. This achieves efficient video frame reading and alleviates system memory read / write pressure.

[0143] Optionally, refer to Figure 3 A multi-terminal video collaboration method applied to the decoding end may include: network card reception, wherein bypass frames are stored in high-bit memory, the main bitstream is decoded and stored in low-bit memory and shared memory, and VD and HAL are read from high-bit memory and shared memory to retrieve video frames for use by the APP.

[0144] It should be noted that the specific implementation of the multi-terminal video collaboration method from the perspective of the acquisition end in this application embodiment can be found in the description of the multi-terminal video collaboration method from the perspective of the acquisition end above, and will not be repeated here.

[0145] In an exemplary embodiment, step 304, parsing the data packets based on frame type to obtain video frames written to different memory regions, includes:

[0146] If the frame type of the data packet is the main bitstream, then the data packet is parsed based on the video decoder, video frames are generated and written to shared memory;

[0147] If the data packet's frame type is a bypass frame, the data packet is parsed based on the bypass parser, a video frame is generated, and written to high-order memory.

[0148] Specifically, if the frame type of the data packet is the main bitstream, the data packet is parsed based on the video decoder to generate video frames and write them to shared memory; if the frame type of the data packet is a bypass frame, the data packet is parsed based on the bypass parser to generate video frames and write them to high-order memory, thereby realizing video stream reception and classification processing.

[0149] For example, refer to Figure 5 Video stream reception and classification processing can include: monitoring camera data through a network packet receiving module to receive main stream and bypass frame data packets; parsing the frame type based on the data header identifier through a network frame parsing module, and sending the two types of data to different processing paths, either video frame decoding or bypass frame parsing. The video frame decoding path includes: main stream → decoder → decoded frame → written to shared memory. Figure 5The memory in the memory is copied to shared memory. The bypass frame parsing path includes: bypass frame (MJPEG / YUV) → bypass parser → direct write to high-order memory. When the bypass parser successfully parses the frame, it notifies VD of the frame scheduling policy, such as the sequence number column.

[0150] In an exemplary embodiment, step 306, obtaining corresponding video frames from different memory regions based on a frame scheduling strategy, includes:

[0151] Instruct the memory manager to record frame information of data packets written to the high-order memory.

[0152] The virtual driver schedules the frame data of the corresponding video frame from shared memory or high-order memory based on the frame information for the hardware abstraction layer to read.

[0153] Specifically, by having the memory manager record the frame information of data packets written to high memory, the virtual driver can schedule the corresponding video frame data from shared memory or high memory based on the frame information for the hardware abstraction layer to read. This enables unified abstraction of frame data and access by the virtual driver, as well as real-time frame consumption by the hardware abstraction layer, such as the HAL layer and the application.

[0154] For example, the unified abstraction of frame data and access to the virtual driver use the virtual driver (VD) module as an access bridge between the HAL layer and the underlying frame data, referencing... Figure 5 Specifically, this can include: The VD schedules corresponding frame data from shared memory or high-order memory according to a frame scheduling strategy, without the APP needing to be aware of the frame source. The VD integrates decoded frames and bypass frames, providing memory addresses to the HAL layer. This enables real-time frame consumption by the HAL layer and the APP. Specifically, the HAL layer uniformly exposes an image frame reading interface, supporting frame-by-frame / stream retrieval, thus completing... Figure 5 The HAL layer reads data from the application layer. The application layer (such as motion-sensing games, AR motion analysis, etc.) obtains video frames in real time through the HAL interface for business processing, thus completing the process. Figure 5 The application is presented in the middle. Through the unified memory management and control of the virtual driver VD and HAL layer, the bypass frame parsing module and VD complete the indexing of bypass frames and main code stream. The VD and HAL layers uniformly complete the high and low memory management. The upper layer application does not need to be aware of memory changes and can directly access the bypass frame memory to complete efficient reading, which alleviates the system memory read and write pressure.

[0155] To further illustrate the solution of this application, a specific example is provided below. Optionally, when performing AR motion-related multi-terminal collaborative services, this embodiment of the application triggers and runs a bypass frame mechanism according to the following steps to reduce the decoding and memory read / write pressure on the TV end while ensuring real-time performance. The specific steps include:

[0156] Step 1, Service Startup and Regular Transmission:

[0157] When a user launches an AR motion-sensing application on their TV, the TV initiates a conversation with the camera (or the camera detects a subscription).

[0158] The camera continuously acquires images (RAW / YUV), encodes them in H.264 / H.265 main bitstream as originally set, and sends them to the TV via RTP / UDP; at the same time, the camera activates the RTCP / bandwidth sensing mechanism, periodically receiving / sending RTCP packets to estimate link bandwidth and packet loss rate.

[0159] Step 2, Performance / Bandwidth Assessment and Bypass Policy Triggering:

[0160] The camera or control module calculates the current available uplink bandwidth R_avail based on RTCP messages, receiver reports (packet loss, jitter) and local measurements, and subtracts the reserved ratio β (e.g., 20%) to obtain the effective available bandwidth B_usable = (1-β)·R_avail.

[0161] The TV simultaneously reports local performance metrics (such as CPU utilization, memory bandwidth usage, and decoding queue latency). When it is detected that there are enough B_usables to allocate to bypass frames and the TV's decoding or memory load is close to the threshold, the bypass frame transmission strategy is triggered.

[0162] Step 3, Bypass bandwidth budget calculation (example formula):

[0163] Let the target bitrate of the main bitstream be R_main (fixed by the encoder or limited by QoS, and not occupying bypass bandwidth).

[0164] Bypass allocable bandwidth: B_BYP = B_usable - R_main.

[0165] If the bypass size per frame is S_bypass (KB), then the bypass frame rate is:

[0166]

[0167] Here, B_BYP is the available bandwidth for bypass frames, i.e., the network transmission rate, which needs to have a margin (e.g., reserving 20% ​​for packet loss or buffering), and the unit is the same as R_BYP. R_BYP is the bit rate required for bypass frames, i.e. the amount of data per second, and the unit is Kbps / Mbps. The unit of data amount S is KB (bytes). To convert Kbps, you need to divide by 8. 1024KB = 1MB, so divide by 1024.

[0168] Assuming R_avail=20Mbps, β=0.2 → B_usable=16Mbps; S_I=220KB, S_P=30KB, F=30fps, GOP=15, R_main≈12Mbps → B_BYP=4Mbps; if S_bypass=200KB, then F_BYP≈2.5fps (approximately 2~3 bypass images per second, each GOP contains 1~2 bypass frames).

[0169] Step 4, Camera Encoding and Frame Scheduling:

[0170] The frame scheduling module determines the bypass insertion strategy (inserting one MJPEG frame for each GOP) based on the F_BYP and GOP strategy (GOP=15). For frames selected for bypass, the camera outputs in parallel, still retaining fragments of the main bitstream (to ensure display continuity). A bypass frame (MJPEG) is synchronously generated and encapsulated, a custom identifier (header + type) is added before the UDP payload, and then it is sent.

[0171] Step 5: TV reception, parsing, and storage:

[0172] The network receiving module checks the custom identifier of the arriving UDP packet: if it is a bypass frame (type=MJPEG), it is sent to the bypass parser; otherwise, it is processed according to the normal RTP decoding process.

[0173] The bypass parser writes bypass frames directly to high-order memory (bypassing the low-order decoder output area and shared memory copy), and registers their timestamps / sequence numbers and source-to-frame index tables (e.g., I,P,P,P,P,P,P,P,M,P,P,P,P,P,P,P) in the memory manager; the main stream frames are still decoded by the video decoder and written to shared memory.

[0174] The Virtual Driver (VD) maintains a unified frame index table and schedules frames from shared memory or high-order memory for the HAL layer to read based on timestamps / policies, so that upper-layer applications do not need to identify the frame source.

[0175] Step 6: APP Consumption and System Loop Monitoring:

[0176] The APP (motion sensing / AR) obtains low-latency bypass frames and decoded frames of the main bitstream through the HAL interface for computational tasks such as image recognition and pose estimation. Since index values ​​are maintained at the VD position, applications above the HAL layer do not need to be aware of memory blocks.

[0177] The camera continuously monitors RTCP metrics and TV performance; when bandwidth decreases or TV load recovers to a safe threshold, the camera automatically stops sending bypass frames or reduces the bypass frequency, reverting to main stream-only mode.

[0178] If one P-frame is replaced or provided by a bypass frame within a GOP (15 frames) (i.e., this bypass frame is directly used by the APP without being written to shared memory), the reduction in the number of frames that need to be written to shared memory and read by VD is approximately 1 / 15 = 6%, and the corresponding number of memory copy operations and bus bandwidth will decrease by at least 6%. With the increase in bandwidth environment (such as Wi-Fi 6, FTTR), the actual copy performance will continue to improve.

[0179] In one exemplary embodiment, refer to Figure 4 This application also provides a multi-terminal video collaboration system for implementing the aforementioned multi-terminal video collaboration method. Specifically, this system is applicable to smart home environments where cameras and television terminals (set-top boxes) are connected to the same local area network (LAN) via a home gateway or router, constructing a multi-terminal collaborative video processing system for real-time services such as motion-sensing games, AR motion, and behavior analysis. The core idea is to significantly reduce the decoding and memory load on the television terminal while ensuring service smoothness through a dual-path video frame transmission mechanism and frame scheduling perception algorithm. The system architecture mainly consists of two parts: the camera terminal and the television terminal. The system architecture diagram is shown below. Figure 4 .

[0180] The camera mainly includes: (1) Image acquisition module: acquires raw video frames (RAW) from the image sensor as the starting point of the entire video processing link. (2) Preprocessing module: performs preprocessing operations such as brightness adjustment, noise reduction, and color correction on the raw image frames to improve image quality and compression efficiency. (3) YUV subsampling module: performs subsampling processing on the image frames (such as 4:2:0 or 4:2:2) to compress the image volume for video encoding and generating bypass transmission frames. (4) MJPEG encoding module: encodes selected frames (periodic frames) into MJPEG format as bypass frames. (5) Video encoder: performs standard H.264 / H.265 encoding on the continuous video stream to form the main bit stream. (6) Bandwidth awareness module: evaluates the available bandwidth of the current network in real time based on mechanisms such as RTCP feedback and UDP feedback, and provides it for the scheduling strategy module to call. (7) Frame scheduling awareness module: Based on factors such as service type, network status, and GOP structure, it intelligently decides whether to send bypass frames (YUV frames / MJPEG frames) in parallel and dynamically controls the bypass frame ratio. (8) Network transmission module: It packages and transmits the main stream and bypass frame data to the TV end via RTP over UDP or TCP, and has in-band differentiation capability.

[0181] The TV terminal mainly includes: (1) Network receiving module: listens to the data frames sent by the camera in the local area network and supports the synchronous reception of the main stream and the bypass stream. (2) Network frame parsing module: identifies and parses the transmission data packets (RTP frames), distinguishes the video stream frame type (I / P / B frames or bypass frames) and the source channel. (3) Bypass frame parser: quickly parses the received MJPEG or YUV bypass frames and writes them into the high-order memory area, skipping the video decoder. (4) Video decoder: decodes the standard H.264 / H.265 frame data in the main stream for regular video display or multi-frame fusion operations. (5) Memory manager: uniformly manages the shared memory and high-order memory space, and coordinates the storage and access permissions of the main stream decoded frames and bypass frames. (6) Virtual driver: encapsulates the access path of the main stream and bypass frames, provides a unified frame access interface to the HAL layer, and shields the upper layer application from the perception of the frame source. (7) HAL layer / APP: Access image frame data through standard HAL interface, supporting real-time business scenarios such as motion-sensing games, AI recognition, and AR interaction.

[0182] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0183] Based on the same inventive concept, this application also provides a multi-terminal video collaboration device for implementing the multi-terminal video collaboration method applied to the acquisition end as described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more multi-terminal video collaboration device embodiments provided below can be found in the limitations of the multi-terminal video collaboration method described above, and will not be repeated here.

[0184] In one exemplary embodiment, such as Figure 6 As shown, a multi-terminal video collaboration device 800 for use at the acquisition end is provided, including: an acquisition unit 801, a bandwidth sensing unit 802, and an encoding and transmission unit 803, wherein:

[0185] The acquisition unit 801 is used to acquire continuous image frames and to acquire the main bitstream and bypass frames based on the continuous image frames.

[0186] The bandwidth sensing unit 802 is used to output the main bit stream and obtain network bandwidth status.

[0187] The encoding and transmission unit 803 is used to determine the transmission strategy for bypass frames based on network bandwidth conditions and encapsulate them into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

[0188] In one embodiment, the encoding transmission unit 803 is further configured to: determine whether to trigger the transmission of a bypass frame based on the available bandwidth; determine the allocable bandwidth of the bypass frame based on the target bit rate of the main bitstream; calculate the frequency of the bypass frame based on the allocable bandwidth and the single frame size of the bypass frame; and encapsulate the bypass frame into a data packet based on the frequency of the bypass frame and transmit it.

[0189] In one embodiment, the encoding transmission unit 803 is further configured to: if the main bitstream and the bypass frame share the same transmission channel for transmission, distinguish the main bitstream and the bypass frame based on the identification field.

[0190] In one embodiment, the acquisition unit 801 is further configured to: acquire continuous image frames through the image acquisition module of the acquisition end; preprocess and color compress the continuous image frames; generate a standard main bitstream based on the continuous image frames through the video encoder of the acquisition end, and select some key frames and / or periodic frames of the continuous image frames to form bypass frames.

[0191] In one exemplary embodiment, refer to Figure 7 A multi-terminal video collaboration device 900 for use in decoding is provided, comprising: a receiving unit 901, a parsing unit 902, and a video reading unit 903, wherein:

[0192] The receiving unit 901 is used to receive data packets and determine the frame type of the data packets. The frame type includes the main stream and bypass frames.

[0193] The parsing unit 902 is used to parse data packets based on frame type and obtain video frames written to different memory regions; the memory regions include at least the memory regions that do not go through the video decoder.

[0194] The video reading unit 903 is used to obtain corresponding video frames from different memory regions based on a frame scheduling strategy.

[0195] In one embodiment, the parsing unit 902 is further configured to: if the frame type of the data packet is a main bitstream, then parse the data packet based on the video decoder, generate a video frame and write it to the shared memory; if the frame type of the data packet is a bypass frame, then parse the data packet based on the bypass parser, generate a video frame and write it to the high-order memory.

[0196] In one embodiment, the video reading unit 903 is further configured to instruct the memory manager to record the frame information of the data packets written to the high-order memory; and to instruct the virtual driver to schedule the frame data of the corresponding video frame from the shared memory or the high-order memory for the hardware abstraction layer to read based on the frame information.

[0197] Each module in the aforementioned multi-terminal video collaboration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0198] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a multi-terminal video collaboration method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0199] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0200] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method embodiments.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0206] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-terminal video collaboration method, characterized in that, Applied to the acquisition end, the method includes: Acquire consecutive image frames, and obtain the main bitstream and bypass frames based on the consecutive image frames; Output the main bitstream and obtain network bandwidth status; Based on the network bandwidth conditions, a transmission strategy is determined for the bypass frames, and they are encapsulated into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

2. The method according to claim 1, characterized in that, The step of determining a transmission strategy for the bypass frame based on the network bandwidth conditions and encapsulating it into a data packet for transmission includes: Whether to trigger the transmission of the bypass frame is determined based on the available bandwidth. The allocable bandwidth of the bypass frame is determined based on the target bitrate of the main bitstream, and the frequency of the bypass frame is calculated based on the allocable bandwidth and the single frame size of the bypass frame. The bypass frames are encapsulated into data packets based on their frequency and then transmitted.

3. The method according to claim 1, characterized in that, The method further includes: If the main stream and the bypass frame share the same transmission channel for transmission, the main stream and the bypass frame are distinguished based on the identification field.

4. The method according to claim 1, characterized in that, The step of acquiring continuous image frames, and acquiring the main bitstream and bypass frames based on the continuous image frames, includes: The continuous image frames are acquired by the image acquisition module of the acquisition terminal; The consecutive image frames are preprocessed and color compressed; The video encoder at the acquisition end generates a standard main bitstream based on the continuous image frames, and selects some key frames and / or periodic frames of the continuous image frames to form the bypass frames.

5. A multi-terminal video collaboration method, characterized in that, Applied to the decoding end, the method includes: Receive data packets and determine the frame type of the data packets, the frame type including main stream and bypass frames; The data packet is parsed based on the frame type to obtain video frames written to different memory regions; the memory regions include at least those that do not pass through the video decoder. The corresponding video frames are obtained from the different memory regions based on the frame scheduling strategy.

6. The method according to claim 5, characterized in that, The step of parsing the data packet based on the frame type to obtain video frames written to different memory regions includes: If the frame type of the data packet is a main bitstream, the data packet is parsed based on the video decoder to generate video frames and write them to shared memory; If the frame type of the data packet is a bypass frame, the data packet is parsed based on the bypass parser to generate a video frame and write it to the high-order memory.

7. The method according to claim 6, characterized in that, The step of obtaining the corresponding video frames from the different memory regions based on the frame scheduling strategy includes: Instruct the memory manager to record the frame information of the data packets written to the high-order memory; The virtual driver, based on the frame information, schedules the frame data of the corresponding video frame from the shared memory or the high-order memory for the hardware abstraction layer to read.

8. A multi-terminal video collaboration device, characterized in that, The device, applied to the data acquisition end, includes: The acquisition unit is used to acquire continuous image frames and acquire the main bitstream and bypass frames based on the continuous image frames; A bandwidth sensing unit is used to output the main bitstream and obtain network bandwidth status. An encoding and transmission unit is used to determine a transmission strategy for the bypass frames based on the network bandwidth conditions, and encapsulate them into data packets for transmission; the transmission strategy includes determining the proportion of bypass frames.

9. A multi-terminal video collaboration device, characterized in that, The device, applied at the decoding end, includes: A receiving unit is configured to receive data packets and determine the frame type of the data packets, wherein the frame type includes the main stream and bypass frames; The parsing unit is used to parse the data packet based on the frame type to obtain video frames written to different memory regions; the memory regions include at least the memory regions that do not go through the video decoder; The video reading unit is used to obtain the corresponding video frames from the different memory regions based on the frame scheduling strategy.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images