A multi-terminal cooperative digital media data processing system and method

Abstract

The application provides a multi-terminal cooperative digital media data processing system and method, and relates to the technical field of multi-terminal cooperation. Each member terminal generates a local media processing checkpoint data packet in a local audio and video processing process. Each member terminal sends the generated checkpoint data packet to a master terminal. The master terminal generates a global consistency checkpoint mark of a checkpoint frame interval after confirming that each member terminal has submitted the checkpoint data packet of the same checkpoint frame interval. After offline reconnection, the member terminal inquires the master terminal about the last valid global consistency checkpoint mark. The reconnected member terminal recovers the state configuration of the audio and video encoder at the checkpoint time and the local rendering buffer according to the global consistency checkpoint mark. The application can realize consistent recovery of the digital media processing state after multi-terminal offline reconnection, thereby ensuring the uniformity of the media content time sequence of cooperative processing.

Description

Technical Field

[0001] This application relates to the field of multi-terminal collaboration technology, and more specifically, to a multi-terminal collaborative digital media data processing system and method. Background Technology

[0002] With the continuous enrichment of digital media processing scenarios, multi-terminal collaboration technology has become an important technical means to achieve efficient media processing. This technology establishes collaborative sessions and divides processing task areas through a master terminal, enabling multiple terminals to complete audio and video processing in parallel. It effectively breaks through the limitations of single-terminal computing power and storage, improves the parallel efficiency of media processing, and can be widely used in scenarios such as collaborative editing, distributed transcoding, and multi-terminal synchronous rendering, providing efficient technical support for digital media collaborative processing businesses.

[0003] In existing multi-terminal collaborative digital media processing, terminals are susceptible to offline reconnection due to network fluctuations and device malfunctions. Current technology lacks an effective mechanism for backing up media processing status and global synchronization. After reconnection, locally cached processing progress, intermediate encoding data, and rendering status are easily lost or deviated, making it difficult to quickly align with the global processing progress. Inconsistent status between terminals directly leads to disordered timing and content mismatch in output media segments, seriously affecting the stability of collaborative processing and media output quality, and limiting the practical application of multi-terminal collaborative technology. Therefore, how to achieve consistent restoration of digital media processing status after offline reconnection of multiple terminals, thereby ensuring the consistency of media content timing in collaborative processing, has become a challenge for the industry. Summary of the Invention

[0004] This application provides a multi-terminal collaborative digital media data processing system and method, which can realize the consistent restoration of the digital media processing status after multiple terminals reconnect offline, thereby ensuring the time sequence consistency of the collaboratively processed media content.

[0005] In a first aspect, this application provides a multi-terminal collaborative digital media data processing method, which includes the following steps:

[0006] The master terminal establishes a collaborative processing session and allocates processing task intervals to each member terminal. During the local audio and video processing, when the processing progress exceeds the preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing.

[0007] Each member terminal sends the generated checkpoint data packet to the main control terminal, which then indexes and stores the checkpoint data packets from each member terminal.

[0008] After confirming that all member terminals have submitted checkpoint data packets with the same checkpoint frame interval, the master terminal generates a global consistency checkpoint flag for that checkpoint frame interval and broadcasts the global consistency checkpoint flag to all member terminals.

[0009] After reconnecting offline, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the state configuration of the audio and video encoder at the checkpoint time and the local rendering buffer according to the global consistency checkpoint marker.

[0010] In this embodiment, the main control terminal establishes a collaborative processing session and allocates processing task ranges to each member terminal, specifically including:

[0011] The main control terminal obtains the device identifiers and capability information of all member terminals participating in the collaborative processing. The capability information includes the audio and video encoding capabilities, rendering buffer capacity, and computing resource reserves of each member terminal.

[0012] Based on the capability information of all member terminals, the digital media data to be processed is divided into multiple processing task ranges;

[0013] Based on the processing capacity and current load status of each member terminal, a processing task range is allocated to each member terminal, and a task allocation table is generated.

[0014] The master terminal sends a session establishment instruction to each member terminal. The session establishment instruction includes the allocated processing task range, the preset checkpoint frame interval, and the communication address of the master terminal.

[0015] In this embodiment, when the processing progress of each member terminal exceeds the preset checkpoint frame interval during local audio and video processing, the generation of a checkpoint data packet for local media processing specifically includes:

[0016] After receiving the session establishment instruction, each member terminal initializes its local audio and video encoder and rendering buffer, and begins audio and video processing according to the assigned processing task range.

[0017] Each member terminal monitors the local audio and video processing progress in real time and compares the number of processed frames with the preset checkpoint frame interval.

[0018] When the processing progress exceeds the preset checkpoint frame interval, the member terminal pauses the current processing task and obtains the status configuration parameters of the audio and video encoder at the checkpoint moment and the current data snapshot of the local rendering buffer.

[0019] The member terminal encapsulates the acquired status configuration parameters and current data snapshot into a checkpoint data packet, and marks the member terminal's identifier, current checkpoint frame interval number, and timestamp in the checkpoint data packet.

[0020] In this embodiment, each member terminal sends the generated checkpoint data packet to the main control terminal. The main control terminal indexes and stores the checkpoint data packets of each member terminal, specifically including:

[0021] After generating a checkpoint data packet for local media processing, each member terminal sends the checkpoint data packet to the main control terminal.

[0022] The master terminal receives checkpoint data packets from different member terminals and parses out the member terminal identifier, checkpoint frame interval number and timestamp from each checkpoint data packet;

[0023] The master control terminal establishes a hierarchical index structure with the checkpoint frame interval sequence number as the main index and the member terminal identifier as the sub-index.

[0024] The master terminal stores each checkpoint data packet in the storage location corresponding to the hierarchical index structure, and records the reception time and storage path of each checkpoint data packet.

[0025] In this embodiment, after confirming that all member terminals have submitted checkpoint data packets with the same checkpoint frame interval, the master terminal generates a global consistency checkpoint marker for that checkpoint frame interval and broadcasts the global consistency checkpoint marker to all member terminals. Specifically, this includes:

[0026] The master terminal uses the checkpoint frame interval sequence number as the polling period to traverse the hierarchical index structure and submit the checkpoint data packet submission notification for each member terminal under that sequence number.

[0027] When the master terminal detects that there is a checkpoint frame interval where a checkpoint data packet of a certain member terminal is missing, it starts an asynchronous waiting timer and initiates a checkpoint data packet retransmission request for the missing member terminal.

[0028] After confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision-maker to generate the consistency arbitration certificate for the checkpoint frame interval.

[0029] The master terminal generates a timestamped global consistency checkpoint marker based on the consistency arbitration certificate, the current global clock reference, and the digital digest of the checkpoint data packets of all member terminals at this checkpoint frame interval.

[0030] The master terminal synchronously distributes the global consistency checkpoint mark to all member terminals and writes the global consistency checkpoint mark into the master terminal's persistent checkpoint log.

[0031] In this embodiment, after confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision maker to generate the consistency arbitration certificate for the checkpoint frame interval. Specifically, this includes:

[0032] The master terminal sorts the checkpoint data packets of all member terminals that have been confirmed to have completed checkpoints within the target checkpoint frame interval according to the lexicographical order of the member terminal identifiers, forming an ordered linked list of checkpoint data packets.

[0033] Traverse the ordered checkpoint data packet linked list, extract the media processing status hash value of the member terminal in the frame interval from each checkpoint data packet, and aggregate all status hash values ​​in the linked list order to generate an aggregated hash chain.

[0034] The target checkpoint frame interval number and the root hash value of the aggregated hash chain are input into the global consistency decision-maker, which compares and matches the root hash value with the locally stored standard hash template.

[0035] After the comparison and matching are successful, the master terminal binds and signs the target checkpoint frame interval sequence number, root hash value and current global consistency counter value to generate a consistency arbitration certificate for that checkpoint frame interval.

[0036] In this embodiment, after offline reconnection, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the audio / video encoder's state configuration and local rendering buffer at the checkpoint time based on the global consistency checkpoint marker, specifically including:

[0037] After re-establishing the communication link with the main control terminal, the member terminal sends a recovery query request to the main control terminal, carrying its own terminal identifier and the sequence number of the last confirmed checkpoint frame interval before going offline.

[0038] The master terminal responds to the recovery query request by tracing back the most recent valid global consistency checkpoint marker and its associated checkpoint frame interval number from the persistent checkpoint log, and returns it to the reconnecting member terminal.

[0039] The reconnecting member terminal initiates a state recovery request for the checkpoint frame interval corresponding to the received global consistency checkpoint flag to the master terminal, and the request carries its own terminal identifier.

[0040] According to the state recovery request, the main control terminal extracts the checkpoint data packet of the member terminal at the target checkpoint frame interval from the hierarchical index structure, and encapsulates the audio and video encoder state configuration parameters and rendering buffer data snapshot in the checkpoint data packet into a state recovery response;

[0041] The reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio and video encoder, and remaps the rendering buffer data snapshot to the local rendering buffer.

[0042] In this embodiment, the reconnecting member terminal initiates a state recovery request for the checkpoint frame interval corresponding to the received global consistency checkpoint flag to the master terminal. The request carries its own terminal identifier, specifically including:

[0043] The reconnecting member terminal parses the target checkpoint frame interval number and the corresponding consistency arbitration certificate from the received global consistency checkpoint marker, and extracts the last confirmed checkpoint frame interval number before going offline from the local cache.

[0044] The parsed target checkpoint frame interval number is compared with the last confirmed checkpoint frame interval number before offline by step difference. When the step difference is greater than the preset threshold, an incremental recovery identifier is generated.

[0045] The terminal identifier, target checkpoint frame interval sequence number, consistency arbitration certificate, and incremental recovery identifier are encapsulated into a state recovery request message;

[0046] The reconnected member terminal sends the status recovery request message to the recovery request processing queue of the master terminal and starts a recovery response waiting timer.

[0047] In this embodiment, the reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio / video encoder, and simultaneously remaps the rendering buffer data snapshot to the local rendering buffer. Specifically, this includes:

[0048] The reconnected member terminal parses the encoder state configuration parameters, render buffer data snapshot, and response verification sequence number from the received state recovery response, and performs a matching verification between the response verification sequence number and the local recovery request sequence number.

[0049] Based on the encoder status configuration parameters after the matching verification is passed, the address space of the local audio and video encoder status register is traversed field by field, and the value of each field in the encoder status configuration parameters is written to the corresponding status register unit according to the register offset address.

[0050] The render buffer data snapshot is deserialized into raw pixel data blocks, and the target mapping area of ​​the local render buffer is located based on the buffer layout descriptor carried in the snapshot.

[0051] The reconnected member terminal copies the deserialized original pixel data block to the physical memory page of the target mapping region and refreshes the address mapping table of the rendering buffer to complete the remapping.

[0052] Secondly, this application provides a multi-terminal collaborative digital media data processing system for executing a multi-terminal collaborative digital media data processing method, the digital media data processing system comprising:

[0053] The checkpoint data packet generation module is used by the master terminal to establish a collaborative processing session and allocate processing task intervals to each member terminal. When the processing progress of each member terminal exceeds the preset checkpoint frame interval during local audio and video processing, a checkpoint data packet for local media processing is generated.

[0054] The checkpoint data packet index storage module is used by each member terminal to send the generated checkpoint data packets to the main control terminal, and the main control terminal indexes and stores the checkpoint data packets of each member terminal.

[0055] The global consistency checkpoint marking module is used by the master terminal to generate a global consistency checkpoint mark for the same checkpoint frame interval after confirming that all member terminals have submitted checkpoint data packets for the same checkpoint frame interval, and broadcast the global consistency checkpoint mark to all member terminals.

[0056] The state consistency recovery module is used to query the main control terminal for the most recent valid global consistency checkpoint marker after a member terminal reconnects from offline. The reconnected member terminal restores the state configuration of the audio and video encoder and the local rendering buffer at the checkpoint time based on the global consistency checkpoint marker.

[0057] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0058] The master terminal establishes a collaborative processing session and allocates processing task intervals to each member terminal. During local audio and video processing, when the processing progress exceeds a preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing. Each member terminal sends the generated checkpoint data packet to the master terminal, which indexes and stores the checkpoint data packets of each member terminal. After confirming that each member terminal has submitted a checkpoint data packet for the same checkpoint frame interval, the master terminal generates a global consistency checkpoint marker for that checkpoint frame interval and broadcasts the global consistency checkpoint marker to all member terminals. After offline reconnection, a member terminal queries the master terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the state configuration of the audio and video encoder and the local rendering buffer at the checkpoint time based on the global consistency checkpoint marker.

[0059] Therefore, in this application, the audio / video encoder's state configuration and local rendering buffer at the checkpoint can be restored by reconnecting member terminals based on the globally consistent checkpoint marker. First, the master terminal establishes a collaborative processing session and allocates task intervals based on the capabilities of each member terminal. Simultaneously, each member terminal generates a local checkpoint data packet according to a preset checkpoint frame interval, completely preserving a snapshot of the audio / video encoder's state configuration and rendering buffer data, avoiding the loss or deviation of processing progress, intermediate encoding data, and rendering state after offline reconnection. Second, each member terminal sends the checkpoint data packet to the master terminal, which stores it using a hierarchical index structure based on the checkpoint frame interval sequence number and member terminal identifier. This achieves ordered management of multi-terminal state data, ensuring that the terminal can quickly and accurately retrieve its corresponding checkpoint data after reconnection, improving the efficiency of state recovery. Furthermore, the master terminal, after confirming that all member terminals have... After submitting checkpoint data packets at the same frame interval, a global consistency checkpoint marker is generated and broadcast across the network through processes such as consistency arbitration and hash comparison. This establishes a unified global synchronization benchmark for multiple terminals, eliminating the state misalignment problem caused by independent processing by each terminal. Furthermore, after a member terminal reconnects offline, it queries the most recently valid global consistency checkpoint marker, retrieves the corresponding checkpoint data packet to load the encoder state and map the rendering buffer, and combines request-response verification and step difference comparison to achieve accurate state recovery and quickly align the global processing progress. Finally, supplementary mechanisms such as asynchronous waiting for retransmission, persistent log storage, and incremental recovery further ensure the integrity of the global checkpoint and the accuracy of state recovery. This comprehensively solves the problems of output media segment timing disorder and content mismatch caused by inconsistent states between terminals, ensuring the stability of multi-terminal collaborative processing and enhancing the practical application value of multi-terminal collaborative technology.

[0060] In summary, the technical solution adopted in this application can realize the consistent restoration of the digital media processing status after multiple terminals reconnect offline, thereby ensuring the consistency of the timing of collaboratively processed media content. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is an exemplary flowchart of a multi-terminal collaborative digital media data processing method provided in this application;

[0063] Figure 2This is a flowchart illustrating the multi-terminal collaborative digital media processing network scenario provided in this application;

[0064] Figure 3 This is a flowchart illustrating the process of determining the consistency restoration of digital media processing status according to the application provided;

[0065] Figure 4 This is a module structure diagram of a multi-terminal collaborative digital media data processing system provided in this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] This application provides a multi-terminal collaborative digital media data processing system and method. The core of the system involves a master terminal establishing a collaborative processing session and allocating processing task intervals to each member terminal. During local audio and video processing, when the processing progress exceeds a preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing. Each member terminal sends the generated checkpoint data packet to the master terminal, which indexes and stores the checkpoint data packets from each member terminal. After confirming that all member terminals have submitted checkpoint data packets for the same checkpoint frame interval, the master terminal generates a globally consistent checkpoint marker for that checkpoint frame interval and broadcasts the globally consistent checkpoint marker to all member terminals. After offline reconnection, a member terminal queries the master terminal for the most recently valid globally consistent checkpoint marker. The reconnected member terminal restores the audio and video encoder's state configuration and local rendering buffer at the checkpoint time based on the globally consistent checkpoint marker.

[0068] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of a multi-terminal collaborative digital media data processing method according to this embodiment of the present application. The digital media data processing method includes the following steps:

[0069] In step S1, the master terminal establishes a collaborative processing session and allocates processing task intervals to each member terminal. During the local audio and video processing, when the processing progress exceeds the preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing.

[0070] It should be noted that in this application, references Figure 2 As shown in the figure, this is a network scenario diagram of multi-terminal collaborative digital media processing provided in an embodiment of this application. The diagram shows an architecture with a master control terminal as the core and multiple types of member terminals distributed and collaboratively. The master control terminal, as the scheduling and management core of the entire system, has core functions such as session creation, capability assessment, task allocation, and global status control. Specifically, it can be a high-performance server, a professional industrial control host, or a desktop workstation, etc. It establishes a stable connection with several member terminals through communication links such as local area networks, 5G / Wi-Fi, etc. The member terminals include various terminals with audio and video processing capabilities such as computers, mobile phones, and tablets. As specific task execution nodes, they undertake media processing work such as local audio and video encoding, rendering, and compositing.

[0071] In this embodiment, the master terminal establishes a collaborative processing session and allocates processing task ranges to each member terminal, which can be achieved through the following steps:

[0072] The main control terminal obtains the device identifiers and capability information of all member terminals participating in the collaborative processing. The capability information includes the audio and video encoding capabilities, rendering buffer capacity, and computing resource reserves of each member terminal.

[0073] Based on the capability information of all member terminals, the digital media data to be processed is divided into multiple processing task ranges;

[0074] Based on the processing capacity and current load status of each member terminal, a processing task range is allocated to each member terminal, and a task allocation table is generated.

[0075] The master terminal sends a session establishment instruction to each member terminal. The session establishment instruction includes the allocated processing task range, the preset checkpoint frame interval, and the communication address of the master terminal.

[0076] It should be noted that, in this application, the device identifier refers to the identity information used to uniquely distinguish each member terminal, which may be the terminal MAC address or device serial number; the capability information refers to the hardware and software support capabilities of each member terminal for processing audio and video data, wherein the audio and video encoding capability refers to the encoding format and encoding frame rate supported by the member terminal, the rendering buffer capacity refers to the memory space used by the member terminal for temporarily storing rendering data, and the computing resource reserve refers to the CPU and GPU computing power of the member terminal that is not currently occupied; the processing task interval refers to the continuous data segments of the digital media data to be processed, divided according to the frame sequence, and allocated to a single member terminal for independent processing; the task allocation table refers to the association table that records the identifier of each member terminal and the corresponding processing task interval; and the session establishment instruction refers to the instruction sent by the master terminal to the member terminal to start a collaborative processing session.

[0077] In specific implementation, firstly, the master terminal sends a device detection request via LAN UDP broadcast. The request carries the master terminal's own device identifier and the collaborative processing session identifier. Upon receiving the detection request, each member terminal participating in the collaborative processing immediately responds and returns its own device identifier. Simultaneously, it reads its own audio and video encoding capabilities, rendering buffer capacity, and remaining computing resources through system call interfaces. The master terminal collects the response data from all member terminals and uses this response data as the device identifiers and capability information of all member terminals participating in the collaborative processing. Secondly, it first obtains the total number of frames and the data volume per frame of the digital media data to be processed. Based on the audio and video encoding capabilities and remaining computing resources of all member terminals, it determines the processing efficiency weight of each member terminal (the higher the encoding frame rate and the larger the remaining computing resources, the higher the weight; the weight calculation is based on the positive correlation between terminal processing efficiency and encoding frame rate and remaining computing resources). Then, according to the weight, the total number of frames is divided into multiple consecutive processing task intervals. Each interval is marked with a start frame number and an end frame number. The resulting consecutive frame segments are divided into multiple processing task intervals. Next, the master terminal employs a minimum load priority algorithm to obtain the current load status of each member terminal in real time. It prioritizes allocating processing task intervals to member terminals with CPU utilization below the load threshold (where CPU utilization is the core indicator, with the load threshold set at 70%, based on the upper limit of CPU utilization for stable operation of a typical terminal) and matching processing capabilities. This ensures that the load of each member terminal is within a reasonable range. After allocation, the device identifier of each member terminal, the start and end frame numbers of the allocated processing task intervals, and the corresponding capability information are compiled into a structured table, which serves as the task allocation table. Finally, the master terminal uses the TCP protocol to encapsulate the processing task intervals corresponding to each member terminal in the task allocation table, the preset checkpoint frame intervals, and the master terminal's IP address and communication port into a session establishment command. This session establishment command is sent individually to each member terminal to ensure that each member terminal can accurately obtain its corresponding collaborative processing parameters.

[0078] In this embodiment, when the processing progress of each member terminal exceeds the preset checkpoint frame interval during local audio and video processing, the generation of a checkpoint data packet for local media processing can be achieved through the following steps:

[0079] After receiving the session establishment instruction, each member terminal initializes its local audio and video encoder and rendering buffer, and begins audio and video processing according to the assigned processing task range.

[0080] Each member terminal monitors the local audio and video processing progress in real time and compares the number of processed frames with the preset checkpoint frame interval.

[0081] When the processing progress exceeds the preset checkpoint frame interval, the member terminal pauses the current processing task and obtains the status configuration parameters of the audio and video encoder at the checkpoint moment and the current data snapshot of the local rendering buffer.

[0082] The member terminal encapsulates the acquired status configuration parameters and current data snapshot into a checkpoint data packet, and marks the member terminal's identifier, current checkpoint frame interval number, and timestamp in the checkpoint data packet.

[0083] It should be noted that, in this application, the local audio / video encoder refers to the encoding module built into the member terminal for encoding and processing the allocated audio / video data; the rendering buffer refers to the memory area in the member terminal used to temporarily store the data to be rendered during the audio / video processing; the processing progress refers to the number of audio / video frames that the member terminal has completed encoding and rendering; the checkpoint time refers to the specific processing time point when the processing progress reaches or exceeds the checkpoint frame interval; the status configuration parameters refer to the working parameters of the audio / video encoder at the checkpoint time; the current data snapshot refers to a complete copy of all the data to be rendered in the rendering buffer at the checkpoint time; the checkpoint data packet refers to a data packet encapsulating information such as encoder status configuration parameters, rendering buffer data snapshot, and terminal identifier; the checkpoint frame interval sequence number refers to a consecutive number used to distinguish different checkpoint frame intervals, increasing in processing order; and the timestamp refers to the system time at the time the checkpoint data packet is generated, used to mark the data generation sequence.

[0084] It should also be noted that the checkpoint frame interval mentioned in this application refers to the threshold number of frames used by the master terminal to trigger the generation of checkpoint data packets. The checkpoint frame interval can be preset in the following way: The master terminal refers to the general industry standard for real-time audio and video distributed collaborative processing and initially sets the basic preset value of the checkpoint frame interval to 10 frames (the initial value setting is based on balancing the checkpoint triggering frequency and the terminal computing power overhead. 10 frames can ensure the real-time recovery after offline reconnection and will not reduce the terminal processing efficiency due to frequent generation of checkpoint data packets). The master terminal adopts a weighted dynamic calibration algorithm with the average encoding frame rate of member terminals as the core weight parameter. When the encoding frame rate is higher than 30fps, the basic value is lowered to 8 frames, and when the encoding frame rate is lower than 15fps, the basic value is increased to 12 frames. At the same time, the overall system load rate is detected. When the load rate exceeds 70%, the value is increased by 2 frames on the basis of calibration (the load threshold setting is based on the upper limit of computing power for stable terminal operation). The calibrated value is used as the final preset checkpoint frame interval.

[0085] In specific implementation, firstly, each member terminal listens for and receives the session establishment command sent by the master terminal via a TCP port. It parses the processing task range, preset checkpoint frame interval, and master terminal communication address contained in the session establishment command. Then, it calls the local audio / video encoder initialization interface to initialize the parameters of the local audio / video encoder. Simultaneously, it calls the rendering buffer initialization interface to allocate memory space matching its own rendering buffer capacity. After initialization, it reads the raw audio / video data corresponding to the allocated processing task range, calls the encoder encoding interface and rendering interface, and begins audio / video encoding and rendering processing according to the processing task range. Secondly, each member terminal uses a timed sampling monitoring algorithm. After each sampling cycle, it reads the number of processed audio / video frames through the encoder's built-in frame counting interface. It compares the read number of processed frames with the preset checkpoint frame interval in the session establishment command. The comparison uses a successive comparison method; that is, after each sampling, the number of processed frames is compared with the checkpoint frame interval. The comparison result is then used to determine the value of the comparison. The results serve as the basis for monitoring whether the processing progress meets the standards. Next, when the number of processed frames exceeds the preset checkpoint frame interval, the member terminal calls the audio and video processing pause interface to immediately pause the current encoding and rendering processing tasks. Then, through the encoder status reading interface, it reads the status configuration parameters of the audio and video encoder at the checkpoint moment, field by field, including encoding format, frame rate, bit rate, quantization parameters, etc. The set of parameters read is used as the status configuration parameters of the audio and video encoder at the checkpoint moment. At the same time, all the data to be rendered in the local rendering buffer is completely copied to the temporary memory area, and the data copy in the temporary memory area is used as the current data snapshot of the local rendering buffer. Finally, the member terminal calls the data packet encapsulation interface, uses JSON format to encapsulate the obtained status configuration parameters and the current data snapshot as core data into a data packet, and marks its own device identifier, current checkpoint frame interval sequence number and timestamp in the header field of the data packet. The encapsulated data packet is used as the checkpoint data packet for local media processing.

[0086] In step S2, each member terminal sends the generated checkpoint data packet to the master terminal, and the master terminal indexes and stores the checkpoint data packets of each member terminal.

[0087] In this embodiment, each member terminal sends the generated checkpoint data packet to the main control terminal. The main control terminal can index and store the checkpoint data packets of each member terminal using the following steps:

[0088] After generating a checkpoint data packet for local media processing, each member terminal sends the checkpoint data packet to the main control terminal.

[0089] The master terminal receives checkpoint data packets from different member terminals and parses out the member terminal identifier, checkpoint frame interval number and timestamp from each checkpoint data packet;

[0090] The master control terminal establishes a hierarchical index structure with the checkpoint frame interval sequence number as the main index and the member terminal identifier as the sub-index.

[0091] The master terminal stores each checkpoint data packet in the storage location corresponding to the hierarchical index structure, and records the reception time and storage path of each checkpoint data packet.

[0092] In specific implementation, firstly, after each member terminal completes the encapsulation of the checkpoint data packet, it uses a TCP-based reliable data transmission protocol, obtaining the master terminal's IP address and communication port from the session establishment command, to send the encapsulated checkpoint data packet to the master terminal. This transmitted data packet serves as the checkpoint data packet sent to the master terminal. Secondly, the master terminal continuously receives checkpoint data packets from different member terminals through a listening service on a bound port. It uses a JSON parsing interface to deconstruct each received checkpoint data packet, extracting the member terminal's MAC address identifier, checkpoint frame interval sequence number, and generation timestamp from the packet header. These three types of information are used as the core index information of the checkpoint data packet. Next, the master terminal uses a... The hash index construction algorithm uses the checkpoint frame interval number as the hash key to generate a first-level index entry, and then uses the member terminal identifier as the secondary hash key to generate a second-level index branch under the corresponding first-level index entry. This two-layer hash index system is used as a hierarchical index structure, and the hash algorithm used is the MD5 hash algorithm. Finally, the main control terminal divides independent storage blocks in the local solid-state drive storage area according to the correspondence between the first-level index and the second-level index in the hierarchical index structure. Each checkpoint data packet is written into the storage block that matches the corresponding index. At the same time, the system clock interface is called to obtain the system time of the data packet reception, and the physical storage path corresponding to the data packet is obtained through the file system path query interface. The reception time and storage path are used as the auxiliary storage information of the corresponding checkpoint data packet and associated with the hierarchical index structure.

[0093] In step S3, after confirming that all member terminals have submitted checkpoint data packets with the same checkpoint frame interval, the master terminal generates a global consistency checkpoint marker for that checkpoint frame interval and broadcasts the global consistency checkpoint marker to all member terminals.

[0094] In this embodiment, after confirming that all member terminals have submitted checkpoint data packets with the same checkpoint frame interval, the master terminal generates a global consistency checkpoint marker for that checkpoint frame interval and broadcasts the global consistency checkpoint marker to all member terminals. This can be achieved through the following steps:

[0095] The master terminal uses the checkpoint frame interval sequence number as the polling period to traverse the hierarchical index structure and submit the checkpoint data packet submission notification for each member terminal under that sequence number.

[0096] When the master terminal detects that there is a checkpoint frame interval where a checkpoint data packet of a certain member terminal is missing, it starts an asynchronous waiting timer and initiates a checkpoint data packet retransmission request for the missing member terminal.

[0097] After confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision-maker to generate the consistency arbitration certificate for the checkpoint frame interval.

[0098] The master terminal generates a timestamped global consistency checkpoint marker based on the consistency arbitration certificate, the current global clock reference, and the digital digest of the checkpoint data packets of all member terminals at this checkpoint frame interval.

[0099] The master terminal synchronously distributes the global consistency checkpoint mark to all member terminals and writes the global consistency checkpoint mark into the master terminal's persistent checkpoint log.

[0100] It should be noted that, in this application, the polling period refers to the index query period of the master control terminal using the checkpoint frame interval sequence number as the traversal unit; the submission notification refers to the status identifier in the hierarchical index structure that marks the corresponding member terminal as having completed the upload of the checkpoint data packet; the global consistency decision maker refers to the processing unit of the master control terminal used to verify the consistency of checkpoint data across multiple terminals; the consistency arbitration certificate refers to the certificate information generated after consistency verification to prove the legality and validity of the current checkpoint frame interval data; the global clock reference refers to the system clock reference uniformly maintained by the master control terminal for timing synchronization of all terminals; and the global consistency checkpoint mark refers to the identifier information generated by fusing the arbitration certificate, clock reference, and digital digest to mark the global checkpoint as having reached a consensus.

[0101] In specific implementation, firstly, the master control terminal adopts a sequential polling traversal algorithm, using a single checkpoint frame interval sequence number as the single polling cycle, and traverses sequentially according to the first-level index of the hierarchical index structure, querying the submission notification status of the member terminals corresponding to the second-level index under each first-level index; secondly, when the master control terminal identifies a checkpoint frame interval with a missing second-level index during the polling process, it starts an asynchronous waiting timer with a timeout duration of 3 seconds, and simultaneously sends a retransmission request carrying the target checkpoint frame interval sequence number to the corresponding missing member terminal through the TCP communication link; next, after confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision maker to generate a consistency arbitration certificate for the checkpoint frame interval; then, the master control terminal uses SHA256 hashing... The algorithm performs a hash operation on the checkpoint data packets of all member terminals at the current checkpoint frame interval to obtain the corresponding digital digest. It then retrieves the global clock reference maintained by the master terminal and combines and encapsulates the consistency arbitration certificate, the global clock reference information, and all digital digests. Simultaneously, it calls the system clock interface to generate a millisecond-level timestamp and appends it to the encapsulated content. The encapsulation result with the appended timestamp is used as a global consistency checkpoint marker with a timestamp. Finally, the master terminal uses a local area network broadcast transmission method to synchronously distribute the global consistency checkpoint marker to all member terminals participating in the collaborative processing. At the same time, it calls the log writing interface to append the global consistency checkpoint marker, the corresponding checkpoint frame interval sequence number, and the generated timestamp to the persistent checkpoint log file on the local solid-state drive, completing the broadcast and persistent retention operations of the marker.

[0102] In this embodiment, after confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision maker. The generation of the consistency arbitration certificate for the checkpoint frame interval can be achieved through the following steps:

[0103] The master terminal sorts the checkpoint data packets of all member terminals that have been confirmed to have completed checkpoints within the target checkpoint frame interval according to the lexicographical order of the member terminal identifiers, forming an ordered linked list of checkpoint data packets.

[0104] Traverse the ordered checkpoint data packet linked list, extract the media processing status hash value of the member terminal in the frame interval from each checkpoint data packet, and aggregate all status hash values ​​in the linked list order to generate an aggregated hash chain.

[0105] The target checkpoint frame interval number and the root hash value of the aggregated hash chain are input into the global consistency decision-maker, which compares and matches the root hash value with the locally stored standard hash template.

[0106] After the comparison and matching are successful, the master terminal binds and signs the target checkpoint frame interval sequence number, root hash value and current global consistency counter value to generate a consistency arbitration certificate for that checkpoint frame interval.

[0107] It should be noted that, in this application, the target checkpoint frame interval refers to the checkpoint frame interval at which all member terminals have been confirmed to have submitted checkpoint data packets; the lexicographical order refers to the rule of sorting according to the character encoding order of the member terminal identifiers; the checkpoint data packet linked list refers to a linear data structure formed by sequentially concatenating the corresponding checkpoint data packets of each member terminal; the media processing status hash value refers to a fixed-length hash feature string representing the local audio and video processing status recorded in the member terminal checkpoint data packet; the aggregated hash chain refers to a hash chain formed by sequentially aggregating all status hash values ​​according to the linked list order; the root hash value refers to the final hash result at the end of the aggregated hash chain, used to represent the consistency of the processing status of all member terminals; the standard hash template refers to the standard processing status hash template corresponding to each checkpoint frame interval pre-stored by the master terminal, used to verify the consistency of the multi-terminal status; the global consistency counter value refers to the count variable maintained by the master terminal that increments in the order of global consistency checkpoint generation; and the binding signature refers to the operation of associating and encrypting multiple key information through an encryption algorithm.

[0108] In specific implementation, firstly, the master terminal calls the linked list construction interface to sort the checkpoint data packets of all confirmed member terminals within the target checkpoint frame interval according to the ASCII character encoding order of the member terminal identifiers. The linear structure of the sorted and concatenated data packets is then used as an ordered linked list of checkpoint data packets. Secondly, the master terminal uses a loop traversal algorithm to traverse each checkpoint data packet one by one, starting from the head of the ordered linked list. Through the hash value extraction interface, the media processing status hash value of the member terminal within the target frame interval is extracted from the core field of each data packet. After traversal, the Merkle tree aggregation algorithm is used to treat all extracted status hash values ​​as leaf nodes of the Merkle tree in linked list order. Hash aggregation operations are performed layer by layer, and the final root node hash value is used as the root of the aggregated hash chain. The hash value is then used. Next, the main control terminal inputs the target checkpoint frame interval sequence number and the generated root hash value into the global consistency decision maker. The decision maker calls the string exact comparison algorithm to compare the root hash value with the locally pre-stored standard hash template character by character. If all characters are completely identical during the comparison, the match is considered successful; otherwise, the match is considered unsuccessful. Finally, after confirming that the comparison match is successful, the main control terminal calls the RSA asymmetric encryption algorithm to associate and bind the target checkpoint frame interval sequence number, the root hash value, and the current global consistency counter value (the counter is initially set to 1 and automatically increments by 1 for each global consistency checkpoint marker generated, ensuring that the counter value uniquely corresponds to each checkpoint). The bound information is encrypted and signed using the private key, and the complete encrypted and signed data is used as the consistency arbitration certificate for that checkpoint frame interval.

[0109] In step S4, after reconnecting offline, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the state configuration of the audio and video encoder at the checkpoint time and the local rendering buffer according to the global consistency checkpoint marker.

[0110] Preferably, in this embodiment, after offline reconnection, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the audio / video encoder's state configuration and local rendering buffer at the checkpoint time based on the global consistency checkpoint marker, referring to... Figure 3 As shown in the figure, this is a schematic diagram of the process for determining the consistency restoration of digital media processing state in some embodiments of this application. In this embodiment, determining the consistency restoration of digital media processing state can be achieved by the following steps:

[0111] In step S41, after the member terminal re-establishes the communication link with the main control terminal, it sends a recovery query request to the main control terminal, carrying its own terminal identifier and the sequence number of the last confirmed checkpoint frame interval before going offline.

[0112] In step S42, the master terminal responds to the recovery query request, traces back to locate the most recent valid global consistency checkpoint marker and its associated checkpoint frame interval sequence number from the persistent checkpoint log, and returns it to the reconnecting member terminal.

[0113] In step S43, the reconnecting member terminal initiates a state recovery request for the checkpoint frame interval corresponding to the received global consistency checkpoint flag to the master terminal, and the request carries its own terminal identifier.

[0114] In step S44, the master terminal extracts the checkpoint data packet of the member terminal at the target checkpoint frame interval from the hierarchical index structure according to the state recovery request, and encapsulates the audio and video encoder state configuration parameters and rendering buffer data snapshot in the checkpoint data packet into a state recovery response.

[0115] In step S45, the reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio and video encoder, and remaps the rendering buffer data snapshot to the local rendering buffer.

[0116] It should be noted that, in this application, offline reconnection refers to the process of re-establishing a connection after the communication link between the member terminal and the main control terminal is interrupted; the recovery query request refers to the request information sent by the reconnecting member terminal to the main control terminal to obtain the most recently valid global consistency checkpoint marker; the last confirmed checkpoint frame interval sequence number before offline refers to the frame interval number corresponding to the global consistency checkpoint received and confirmed by the member terminal before offline; the persistent checkpoint log refers to the non-volatile log file stored locally by the main control terminal that records all global consistency checkpoint markers; the most recently valid global consistency checkpoint marker refers to the latest generated and verified global consistency checkpoint identifier in the main control terminal; the state recovery request refers to the request sent by the reconnecting terminal to the main control terminal to obtain checkpoint state data to complete the recovery; the state recovery response refers to the response data encapsulated with encoder state parameters and buffer snapshots sent by the main control terminal to the reconnecting terminal; the state register refers to the hardware register in the audio and video encoder used to store working parameters; and the remapping refers to the operation of copying the buffer data snapshot and associating it with the corresponding memory area of ​​the local rendering buffer.

[0117] In specific implementation, firstly, after detecting the restoration of the communication link with the main control terminal, the member terminal immediately initiates a connection request using the TCP protocol. After the connection is successfully established, it calls the request encapsulation interface, encapsulating its own device identifier and the sequence number of the last confirmed checkpoint frame interval before offline into a recovery query request in JSON format. This encapsulated request is then sent to the main control terminal as a recovery query request. Secondly, after receiving the recovery query request through the port, the main control terminal calls the JSON parsing interface to decompose the terminal identifier and the sequence number before offline in the request. Using a reverse traversal algorithm, it backtracks from the persistent checkpoint log to find the most recently generated globally consistent checkpoint marker with a sequence number greater than or equal to the sequence number before offline. Simultaneously, it extracts the checkpoint frame interval sequence number associated with this marker, encapsulates the marker and the corresponding sequence number into response data, and sends this as the query result returned to the reconnecting member terminal via the TCP link. Then, the reconnecting member terminal... Based on the received global consistency checkpoint marker, a state recovery request is initiated to the master terminal for the checkpoint frame interval corresponding to the marker, carrying its own terminal identifier. Then, after receiving the state recovery request, the master terminal parses out the terminal identifier and the target checkpoint frame interval sequence number. Using the previously established hierarchical index structure (with the checkpoint frame interval sequence number as the main index and the terminal identifier as the sub-index), it calls the index query interface to quickly locate the checkpoint data packet of the member terminal at the target frame interval, extracts the audio and video encoder state configuration parameters and rendering buffer data snapshot from the data packet, encapsulates it in JSON format consistent with the checkpoint data packet, and sends this encapsulated data as a state recovery response to the reconnected member terminal. Finally, the reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the status register of the local audio and video encoder, and remaps the rendering buffer data snapshot to the local rendering buffer.

[0118] In this embodiment, the reconnecting member terminal initiates a state recovery request for the checkpoint frame interval corresponding to the received global consistency checkpoint flag to the master terminal. The request carrying its own terminal identifier can be achieved through the following steps:

[0119] The reconnecting member terminal parses the target checkpoint frame interval number and the corresponding consistency arbitration certificate from the received global consistency checkpoint marker, and extracts the last confirmed checkpoint frame interval number before going offline from the local cache.

[0120] The parsed target checkpoint frame interval number is compared with the last confirmed checkpoint frame interval number before offline by step difference. When the step difference is greater than the preset threshold, an incremental recovery identifier is generated.

[0121] The terminal identifier, target checkpoint frame interval sequence number, consistency arbitration certificate, and incremental recovery identifier are encapsulated into a state recovery request message;

[0122] The reconnected member terminal sends the status recovery request message to the recovery request processing queue of the master terminal and starts a recovery response waiting timer.

[0123] It should be noted that, in this application, the step difference refers to the numerical difference between the target checkpoint frame interval sequence number and the corresponding sequence number before offline; the preset threshold refers to the difference threshold pre-set by the terminal for determining whether incremental recovery is enabled; the incremental recovery identifier refers to a marker field used to inform the master terminal to use incremental method for state recovery; the state recovery request message refers to complete request data encapsulated with terminal identity, target sequence number, arbitration certificate and recovery identifier; the recovery request processing queue refers to the first-in-first-out queue in the master terminal used to cache pending state recovery requests; and the recovery response waiting timer refers to the timing component used by member terminals to wait for the master terminal's recovery response.

[0124] In specific implementation, firstly, the reconnecting member terminal calls the JSON field parsing interface to extract the target checkpoint frame interval sequence number and consistency arbitration certificate from the received global consistency checkpoint marker. Simultaneously, it reads the last confirmed checkpoint frame interval sequence number before offline from its local non-volatile cache area. Secondly, the reconnecting member terminal uses a successive subtraction comparison, performing a difference operation between the parsed target checkpoint frame interval sequence number and the last confirmed checkpoint frame interval sequence number before offline. The calculated result is used as the step difference. The terminal sets a preset threshold of 5 (the threshold is set to balance the transmission overhead of full recovery and incremental recovery; incremental recovery can reduce data transmission when the difference is too large). When the step difference exceeds this preset threshold, a fixed flag bit of 1 is generated in the message field. This flag bit is used as the incremental recovery flag. If the value is not greater than a preset threshold, a fixed flag bit of 0 is generated as an incremental recovery flag. Next, the reconnecting member terminal calls the message encapsulation interface, using the same JSON encapsulation format as the previous checkpoint data packet, and sequentially fills its own terminal MAC address identifier, target checkpoint frame interval sequence number, consistency arbitration certificate, and incremental recovery flag into the message fields. The completed message structure is then used as the state recovery request message. Finally, the reconnecting member terminal writes the state recovery request message into the recovery request processing queue corresponding to the preset port of the master terminal through the TCP communication link. At the same time, it starts a recovery response waiting timer and sets the timer duration to 5 seconds (the duration is set based on the normal maximum delay of the master terminal processing requests and transmitting data within the local area network to avoid infinite waiting). The waiting state after starting the timer is used as the working state before the terminal receives the recovery response.

[0125] In this embodiment, the reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio / video encoder, and simultaneously remaps the rendering buffer data snapshot to the local rendering buffer. This can be achieved through the following steps:

[0126] The reconnected member terminal parses the encoder state configuration parameters, render buffer data snapshot, and response verification sequence number from the received state recovery response, and performs a matching verification between the response verification sequence number and the local recovery request sequence number.

[0127] Based on the encoder status configuration parameters after the matching verification is passed, the address space of the local audio and video encoder status register is traversed field by field, and the value of each field in the encoder status configuration parameters is written to the corresponding status register unit according to the register offset address.

[0128] The render buffer data snapshot is deserialized into raw pixel data blocks, and the target mapping area of ​​the local render buffer is located based on the buffer layout descriptor carried in the snapshot.

[0129] The reconnected member terminal copies the deserialized original pixel data block to the physical memory page of the target mapping region and refreshes the address mapping table of the rendering buffer to complete the remapping.

[0130] It should be noted that, in this application, the response verification sequence number refers to the sequence code attached by the master terminal when sending the status recovery response to verify the legality and uniqueness of the response; the recovery request sequence number refers to the sequence code generated by the reconnecting member terminal when sending the status recovery request, which uniquely corresponds to its own request; the matching verification refers to the verification operation that confirms the status recovery response corresponds to its own request by comparing the response verification sequence number and the recovery request sequence number; the register offset address refers to the address offset corresponding to each parameter field in the status register; the status register unit refers to the smallest storage unit in the status register used to store a single parameter field; and the deserialization refers to... The operation converts the render buffer data snapshot encapsulated in the state recovery response into raw, recognizable pixel data; the raw pixel data block refers to a set of pixel data that can be directly written to the render buffer; the buffer layout descriptor refers to the descriptive information carried in the render buffer data snapshot that describes the memory layout of the local render buffer; the target mapping region refers to a specified memory region in the local render buffer used to receive the data snapshot; the physical memory page refers to the smallest allocation unit of physical memory corresponding to the local render buffer; the address mapping table refers to a table used to record the correspondence between the virtual address of the render buffer and the physical memory address, and refreshing the table can complete the data mapping.

[0131] In specific implementation, firstly, the reconnecting member terminal calls the JSON parsing interface to decompose the received status recovery response, extracting the encoder status configuration parameters, rendering buffer data snapshot, and response verification sequence number one by one. Simultaneously, it reads the recovery request sequence number generated when it sent the status recovery request from the local request cache, comparing the response verification sequence number with the recovery request sequence number. If the two sequence numbers match exactly, the verification is considered successful; otherwise, the verification fails, and the status recovery request is re-initiated. Secondly, after successful verification, the reconnecting member terminal calls the encoder register operation interface, traversing the complete address space of the local audio / video encoder status register according to the preset register address order. It extracts the value of each field from the parsed encoder status configuration parameters and writes the field value one by one into the corresponding status register unit according to the register offset address of each field. Then, the reconnecting... The member terminal uses a JSON deserialization algorithm to deserialize the rendering buffer data snapshot into raw pixel data blocks. Simultaneously, it extracts the buffer layout descriptor from the data snapshot, parsing the starting address, data length, and memory layout of the target mapping region of the local rendering buffer. Using a memory address positioning algorithm, it accurately locates the corresponding target mapping region in the local rendering buffer and uses this located region as the write target for the data snapshot. Finally, the reconnected member terminal copies the deserialized raw pixel data blocks byte-by-byte to the physical memory pages of the target mapping region. After copying, it calls the rendering buffer address mapping refresh interface to update the rendering buffer's address mapping table, resynchronizing the correspondence between virtual addresses and physical memory addresses. The state after data copying and mapping table refresh is taken as the final state of the local rendering buffer after remapping, ensuring that the reconnected member terminal can restore its audio and video processing state to the state at the checkpoint.

[0132] Therefore, in this application, the audio / video encoder's state configuration and local rendering buffer at the checkpoint can be restored by reconnecting member terminals based on the globally consistent checkpoint marker. First, the master terminal establishes a collaborative processing session and allocates task intervals based on the capabilities of each member terminal. Simultaneously, each member terminal generates a local checkpoint data packet according to a preset checkpoint frame interval, completely preserving a snapshot of the audio / video encoder's state configuration and rendering buffer data, avoiding the loss or deviation of processing progress, intermediate encoding data, and rendering state after offline reconnection. Second, each member terminal sends the checkpoint data packet to the master terminal, which stores it using a hierarchical index structure based on the checkpoint frame interval sequence number and member terminal identifier. This achieves ordered management of multi-terminal state data, ensuring that the terminal can quickly and accurately retrieve its corresponding checkpoint data after reconnection, improving the efficiency of state recovery. Furthermore, the master terminal, after confirming that all member terminals have... After submitting checkpoint data packets at the same frame interval, a global consistency checkpoint marker is generated and broadcast across the network through processes such as consistency arbitration and hash comparison. This establishes a unified global synchronization benchmark for multiple terminals, eliminating the state misalignment problem caused by independent processing by each terminal. Furthermore, after a member terminal reconnects offline, it queries the most recently valid global consistency checkpoint marker, retrieves the corresponding checkpoint data packet to load the encoder state and map the rendering buffer, and combines request-response verification and step difference comparison to achieve accurate state recovery and quickly align the global processing progress. Finally, supplementary mechanisms such as asynchronous waiting for retransmission, persistent log storage, and incremental recovery further ensure the integrity of the global checkpoint and the accuracy of state recovery. This comprehensively solves the problems of output media segment timing disorder and content mismatch caused by inconsistent states between terminals, ensuring the stability of multi-terminal collaborative processing and enhancing the practical application value of multi-terminal collaborative technology.

[0133] In summary, the technical solution adopted in this application can realize the consistent restoration of the digital media processing status after multiple terminals reconnect offline, thereby ensuring the consistency of the timing of collaboratively processed media content.

[0134] Example 2: This application provides a multi-terminal collaborative digital media data processing system, referencing... Figure 4 As shown in the figure, this is a block structure diagram of a multi-terminal collaborative digital media data processing system according to this embodiment of the present application. The digital media data processing system includes:

[0135] The checkpoint data packet generation module 100 is used by the master terminal to establish a collaborative processing session, allocate processing task intervals to each member terminal, and generate a checkpoint data packet for local media processing when the processing progress exceeds the preset checkpoint frame interval during the local audio and video processing of each member terminal.

[0136] The checkpoint data packet index storage module 200 is used for each member terminal to send the generated checkpoint data packets to the main control terminal, and the main control terminal to index and store the checkpoint data packets of each member terminal.

[0137] The global consistency checkpoint marking module 300 is used by the master terminal to generate a global consistency checkpoint mark for the same checkpoint frame interval after confirming that all member terminals have submitted checkpoint data packets for the same checkpoint frame interval, and broadcast the global consistency checkpoint mark to all member terminals.

[0138] The processing state consistency recovery module 400 is used to query the main control terminal for the most recent valid global consistency checkpoint mark after the member terminal reconnects offline. The reconnected member terminal restores the state configuration of the audio and video encoder and the local rendering buffer at the checkpoint time according to the global consistency checkpoint mark.

[0139] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A multi-terminal collaborative digital media data processing method, characterized in that, The digital media data processing method includes the following steps: The master terminal establishes a collaborative processing session and allocates processing task intervals to each member terminal. During the local audio and video processing, when the processing progress exceeds the preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing. Each member terminal sends the generated checkpoint data packet to the main control terminal, which then indexes and stores the checkpoint data packets from each member terminal. After confirming that all member terminals have submitted checkpoint data packets with the same checkpoint frame interval, the master terminal generates a global consistency checkpoint flag for that checkpoint frame interval and broadcasts the global consistency checkpoint flag to all member terminals. After reconnecting offline, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the state configuration of the audio and video encoder at the checkpoint time and the local rendering buffer according to the global consistency checkpoint marker.

2. The multi-terminal collaborative digital media data processing method as described in claim 1, characterized in that, The master terminal establishes a collaborative processing session and allocates processing task ranges to each member terminal, specifically including: The main control terminal obtains the device identifiers and capability information of all member terminals participating in the collaborative processing. The capability information includes the audio and video encoding capabilities, rendering buffer capacity, and computing resource reserves of each member terminal. Based on the capability information of all member terminals, the digital media data to be processed is divided into multiple processing task ranges; Based on the processing capacity and current load status of each member terminal, a processing task range is allocated to each member terminal, and a task allocation table is generated. The master terminal sends a session establishment instruction to each member terminal. The session establishment instruction includes the allocated processing task range, the preset checkpoint frame interval, and the communication address of the master terminal.

3. The multi-terminal collaborative digital media data processing method as described in claim 1, characterized in that, During local audio and video processing, when the processing progress exceeds the preset checkpoint frame interval, each member terminal generates a checkpoint data packet for local media processing. This packet specifically includes: After receiving the session establishment instruction, each member terminal initializes its local audio and video encoder and rendering buffer, and begins audio and video processing according to the assigned processing task range. Each member terminal monitors the local audio and video processing progress in real time and compares the number of processed frames with the preset checkpoint frame interval. When the processing progress exceeds the preset checkpoint frame interval, the member terminal pauses the current processing task and obtains the status configuration parameters of the audio and video encoder at the checkpoint moment and the current data snapshot of the local rendering buffer. The member terminal encapsulates the acquired status configuration parameters and current data snapshot into a checkpoint data packet, and marks the member terminal's identifier, current checkpoint frame interval number, and timestamp in the checkpoint data packet.

4. The multi-terminal collaborative digital media data processing method as described in claim 1, characterized in that, Each member terminal sends the generated checkpoint data packet to the main control terminal. The main control terminal indexes and stores the checkpoint data packets from each member terminal, specifically including: After generating a checkpoint data packet for local media processing, each member terminal sends the checkpoint data packet to the main control terminal. The master terminal receives checkpoint data packets from different member terminals and parses out the member terminal identifier, checkpoint frame interval number and timestamp from each checkpoint data packet; The master control terminal establishes a hierarchical index structure with the checkpoint frame interval sequence number as the main index and the member terminal identifier as the sub-index. The master terminal stores each checkpoint data packet in the storage location corresponding to the hierarchical index structure, and records the reception time and storage path of each checkpoint data packet.

5. The multi-terminal collaborative digital media data processing method as described in claim 1, characterized in that, After confirming that all member terminals have submitted checkpoint data packets for the same checkpoint frame interval, the master terminal generates a global consistency checkpoint marker for that checkpoint frame interval and broadcasts the global consistency checkpoint marker to all member terminals. Specifically, the global consistency checkpoint marker includes: The master terminal uses the checkpoint frame interval sequence number as the polling period to traverse the hierarchical index structure and submit the checkpoint data packet submission notification for each member terminal under that sequence number. When the master terminal detects that there is a checkpoint frame interval where a checkpoint data packet of a certain member terminal is missing, it starts an asynchronous waiting timer and initiates a checkpoint data packet retransmission request for the missing member terminal. After confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master control terminal inputs the checkpoint frame interval sequence number into the global consistency decision-maker to generate the consistency arbitration certificate for the checkpoint frame interval. The master terminal generates a timestamped global consistency checkpoint marker based on the consistency arbitration certificate, the current global clock reference, and the digital digest of the checkpoint data packets of all member terminals at this checkpoint frame interval. The master terminal synchronously distributes the global consistency checkpoint mark to all member terminals and writes the global consistency checkpoint mark into the master terminal's persistent checkpoint log.

6. The multi-terminal collaborative digital media data processing method as described in claim 5, characterized in that, After confirming that all member terminals corresponding to the current checkpoint frame interval have submitted checkpoint data packets, the master terminal inputs the checkpoint frame interval sequence number into the global consistency decision maker to generate the consistency arbitration certificate for that checkpoint frame interval, specifically including: The master terminal sorts the checkpoint data packets of all member terminals that have been confirmed to have completed checkpoints within the target checkpoint frame interval according to the lexicographical order of the member terminal identifiers, forming an ordered linked list of checkpoint data packets. Traverse the ordered checkpoint data packet linked list, extract the media processing status hash value of the member terminal in the frame interval from each checkpoint data packet, and aggregate all status hash values ​​in the linked list order to generate an aggregated hash chain. The target checkpoint frame interval number and the root hash value of the aggregated hash chain are input into the global consistency decision-maker, which compares and matches the root hash value with the locally stored standard hash template. After the comparison and matching are successful, the master terminal binds and signs the target checkpoint frame interval sequence number, root hash value and current global consistency counter value to generate a consistency arbitration certificate for that checkpoint frame interval.

7. The multi-terminal collaborative digital media data processing method as described in claim 1, characterized in that, After reconnecting from offline, the member terminal queries the main control terminal for the most recent valid global consistency checkpoint marker. The reconnected member terminal restores the audio and video encoder's state configuration and local rendering buffer at the checkpoint time based on the global consistency checkpoint marker, specifically including: After re-establishing the communication link with the main control terminal, the member terminal sends a recovery query request to the main control terminal, carrying its own terminal identifier and the sequence number of the last confirmed checkpoint frame interval before going offline. The master terminal responds to the recovery query request by tracing back the most recent valid global consistency checkpoint marker and its associated checkpoint frame interval number from the persistent checkpoint log, and returns it to the reconnecting member terminal. The reconnecting member terminal initiates a state recovery request for the checkpoint frame interval corresponding to the received global consistency checkpoint flag to the master terminal, and the request carries its own terminal identifier. According to the state recovery request, the main control terminal extracts the checkpoint data packet of the member terminal at the target checkpoint frame interval from the hierarchical index structure, and encapsulates the audio and video encoder state configuration parameters and rendering buffer data snapshot in the checkpoint data packet into a state recovery response; The reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio and video encoder, and remaps the rendering buffer data snapshot to the local rendering buffer.

8. The multi-terminal collaborative digital media data processing method as described in claim 7, characterized in that, The reconnecting member terminal initiates a state recovery request for the corresponding checkpoint frame interval to the master terminal based on the received global consistency checkpoint marker. The request carries its own terminal identifier, specifically including: The reconnecting member terminal parses the target checkpoint frame interval number and the corresponding consistency arbitration certificate from the received global consistency checkpoint marker, and extracts the last confirmed checkpoint frame interval number before going offline from the local cache. The parsed target checkpoint frame interval number is compared with the last confirmed checkpoint frame interval number before offline by step difference. When the step difference is greater than the preset threshold, an incremental recovery identifier is generated. The terminal identifier, target checkpoint frame interval sequence number, consistency arbitration certificate, and incremental recovery identifier are encapsulated into a state recovery request message; The reconnected member terminal sends the status recovery request message to the recovery request processing queue of the master terminal and starts a recovery response waiting timer.

9. The multi-terminal collaborative digital media data processing method as described in claim 7, characterized in that, The reconnected member terminal parses the state recovery response, loads the parsed encoder state configuration parameters into the state register of the local audio / video encoder, and remaps the rendering buffer data snapshot to the local rendering buffer. Specifically, this includes: The reconnected member terminal parses the encoder state configuration parameters, render buffer data snapshot, and response verification sequence number from the received state recovery response, and performs a matching verification between the response verification sequence number and the local recovery request sequence number. Based on the encoder status configuration parameters after the matching verification is passed, the address space of the local audio and video encoder status register is traversed field by field, and the value of each field in the encoder status configuration parameters is written to the corresponding status register unit according to the register offset address. The render buffer data snapshot is deserialized into raw pixel data blocks, and the target mapping area of ​​the local render buffer is located based on the buffer layout descriptor carried in the snapshot. The reconnected member terminal copies the deserialized original pixel data block to the physical memory page of the target mapping region and refreshes the address mapping table of the rendering buffer to complete the remapping.

10. A multi-terminal collaborative digital media data processing system, used to execute a multi-terminal collaborative digital media data processing method as described in any one of claims 1 to 9, characterized in that, The digital media data processing system includes: The checkpoint data packet generation module is used by the master terminal to establish a collaborative processing session and allocate processing task intervals to each member terminal. When the processing progress of each member terminal exceeds the preset checkpoint frame interval during local audio and video processing, a checkpoint data packet for local media processing is generated. The checkpoint data packet index storage module is used by each member terminal to send the generated checkpoint data packets to the main control terminal, and the main control terminal indexes and stores the checkpoint data packets of each member terminal. The global consistency checkpoint marking module is used by the master terminal to generate a global consistency checkpoint mark for the same checkpoint frame interval after confirming that all member terminals have submitted checkpoint data packets for the same checkpoint frame interval, and broadcast the global consistency checkpoint mark to all member terminals. The state consistency recovery module is used to query the main control terminal for the most recent valid global consistency checkpoint marker after a member terminal reconnects from offline. The reconnected member terminal restores the state configuration of the audio and video encoder and the local rendering buffer at the checkpoint time based on the global consistency checkpoint marker.

Images