Court audio and video multi-terminal fusion forwarding system

Abstract

The application discloses a court audio and video multi-terminal fusion forwarding system, determines a plurality of trial terminals participating in state migration and audio and video forwarding relations before and after state changes of the plurality of trial terminals according to trial state changes, determines candidate switching boundaries according to coding frame states of to-be-switched media streams, determines switching reference time and effective media boundaries corresponding to the to-be-switched media streams in combination with buffer progress and transmission progress, and determines common effective switching boundaries when switching conditions are met; establishes a state migration transaction associated with the plurality of trial terminals and new audio and video forwarding relations thereof, pre-establishes new media forwarding channels before the common effective switching boundaries arrive and keeps old audio and video forwarding relations effective, submits the state migration transaction after the switching conditions of the terminals are met, and realizes cooperative migration of audio and video forwarding relations of the plurality of trial terminals; and reduces terminal state inconsistency and audio and video dislocation in the trial state switching process.

Description

Technical Field

[0001] This invention relates to the field of court trial audio and video processing technology, specifically a court audio and video multi-terminal fusion and forwarding system. Background Technology

[0002] With the development of smart courts and remote trial systems, court proceedings typically involve judge terminals, court clerk terminals, party terminals, witness terminals, trial display terminals, and recording equipment. Media streams from different cameras and audio pickup devices are forwarded to the corresponding terminals according to trial roles and permissions via an audio / video forwarding server. During the trial, as witnesses enter or leave the courtroom, the subjects presenting and examining evidence change, speaking permissions change, and trial stages shift, the audio and video media sources that each terminal needs to receive also change. Therefore, it is necessary to dynamically adjust the audio and video forwarding relationships between multiple terminals.

[0003] Existing audio and video forwarding systems typically modify the media subscriptions or forwarding relationships of each terminal individually based on court control instructions. However, different audio and video media streams have independent encoded frame sequences, and each terminal and forwarding channel has different buffering and transmission progress. When a change in court status simultaneously causes multiple terminals to add, remove, or replace multiple media streams, the encoding boundaries and actual arrival times for each media stream to switch safely are not consistent. If the forwarding relationships are switched item by item according to the arrival order of the control instructions, it is easy for some terminals to have entered the new court status while other terminals remain in the original court status, or for the video of the same terminal to have switched while the corresponding audio is still in the original media stream; at the same time, if the new video stream is not output from a position that can be independently decoded, problems such as short-term inability to decode and abnormal picture may occur.

[0004] In view of this, the present invention provides a court audio and video multi-terminal fusion forwarding system to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a court audio and video multi-terminal fusion forwarding system to solve the technical problems of asynchronous switching of forwarding relationships and audio and video misalignment caused by differences in encoding boundaries, buffer progress and transmission progress of multiple terminals and multiple audio and video when the court hearing status changes.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a court audio and video multi-terminal fusion forwarding system, including a court hearing status management module, a forwarding relationship determination module, a media status detection module, a switching boundary determination module, a status migration transaction management module, a forwarding pre-establishment module, and a collaborative migration module;

[0007] The court hearing status management module is used to acquire court hearing status change events and determine multiple court hearing terminals participating in the status transition based on the court hearing status change events;

[0008] The forwarding relationship determination module is used to determine the old audio and video forwarding relationship of each court terminal before the state change and the new audio and video forwarding relationship after the state change, and to determine the media stream to be switched based on the old audio and video forwarding relationship and the new audio and video forwarding relationship;

[0009] The media state detection module is used to obtain the encoded frame state of each media stream to be switched, and to determine the candidate switching boundary for each media stream to be switched to enter the new audio and video forwarding relationship based on the encoded frame state.

[0010] The switching boundary determination module is used to determine the switching reference time based on the candidate switching boundary, buffer progress and transmission progress of each media stream to be switched, and to determine the effective media boundary corresponding to the switching reference time for each media stream to be switched. When each media stream to be switched can complete the forwarding relationship switching based on the corresponding effective media boundary, the switching reference time is determined as the common effective switching boundary.

[0011] The state transition transaction management module is used to establish a state transition transaction for the court hearing state change event and associate the multiple court hearing terminals and their corresponding new audio and video forwarding relationships with the state transition transaction.

[0012] The forwarding pre-establishment module is used to pre-establish the corresponding media forwarding channel according to the new audio and video forwarding relationship before the common effective switching boundary is reached, and to keep the old audio and video forwarding relationship effective.

[0013] The collaborative migration module is used to submit the state migration transaction when the media forwarding channel corresponding to each court hearing terminal meets the switching conditions and the common effective switching boundary is reached, so that the multiple court hearing terminals are collaboratively migrated from the old audio and video forwarding relationship to the new audio and video forwarding relationship.

[0014] Furthermore, the court hearing status management module stores the terminal media relationship configurations corresponding to different court hearing statuses. The court hearing status change events include at least one of the following: witness entering the court, witness leaving the court, change of the subject presenting evidence, change of the subject of cross-examination, change of speaking authority, and change of court hearing stage.

[0015] The forwarding relationship determination module is used to obtain the corresponding terminal media relationship configuration according to the changed court hearing status, determine the target media source, media type and receiving permissions of each court hearing terminal, and compare it with the terminal media relationship configuration before the status change to determine the media streams that need to be added, kept or removed.

[0016] Furthermore, the media state detection module is used to determine video candidate switching boundaries for the video media stream to be switched based on independently decodeable frames in the video encoded frame sequence; and to determine audio candidate switching boundaries for the audio media stream associated with the video media stream to be switched based on audio frame boundaries and the timestamp relationship between the audio frame boundaries and the corresponding video media stream.

[0017] When the new audio and video forwarding relationship of the same court hearing terminal includes mutually related audio media streams and video media streams, the terminal-level candidate switching boundary corresponding to the court hearing terminal is determined according to the video candidate switching boundary, the audio candidate switching boundary and the preset synchronization tolerance.

[0018] Furthermore, the switching boundary determination module is used to determine the candidate switching time interval based on the terminal-level candidate switching boundaries corresponding to multiple court hearing terminals, and to obtain the buffering progress and transmission progress of each media stream to be switched within the candidate switching time interval;

[0019] Based on the cache progress and transmission progress, determine the remaining processing time required for each media stream to be switched to complete the output of the currently cached and transmitted old media data, and based on the remaining processing time corresponding to each media stream to be switched, determine the switching reference time that satisfies the switching conditions of each media stream to be switched from the candidate switching time interval.

[0020] For each media stream to be switched, a valid media boundary is determined that corresponds to the switching reference time and satisfies the conditions of independent decoding and synchronous output.

[0021] Furthermore, the state transition transaction management module is used to configure a state version identifier for the state transition transaction and associate the state version identifier with the multiple court hearing terminals participating in the state transition and their corresponding new audio and video forwarding relationships;

[0022] The forwarding pre-establishment module is used to detect whether the new media forwarding channel corresponding to each court hearing terminal has been established, and after the new media forwarding channel has been established and the corresponding new media stream has reached the preset caching conditions, the corresponding court hearing terminal is marked as pending switching.

[0023] The collaborative migration module is used to make the new audio and video forwarding relationship corresponding to each court terminal take effect at the common effective switching boundary according to the state version identifier after all the court terminals associated with the state migration transaction are in the state of waiting to switch.

[0024] Furthermore, the state transition transaction management module is used to mark the current common effective switching boundary as invalid when at least one court terminal has not entered the switching state before the current common effective switching boundary is reached;

[0025] The media status detection module is used to reacquire the encoded frame status, buffer progress and transmission progress of the media stream to be switched for the court terminal that has not yet entered the switching state.

[0026] The switching boundary determination module is used to determine the subsequent candidate switching boundary corresponding to the court hearing terminal that has not yet entered the switching state based on the reacquired encoded frame state, buffer progress and transmission progress, and to determine the subsequent terminal-level candidate switching boundary corresponding to other court hearing terminals after the current common effective switching boundary, and to redetermine the common effective switching boundary based on each subsequent terminal-level candidate switching boundary;

[0027] The forwarding pre-establishment module is used to maintain the validity of the old audio and video forwarding relationship corresponding to each court terminal before the newly determined common effective switching boundary is reached.

[0028] Furthermore, the forwarding pre-establishment module is used to establish a corresponding media forwarding channel for the newly added media stream in the new audio and video forwarding relationship before the common effective switching boundary is reached, and to cache media data starting from the starting position that can be independently decoded;

[0029] For the media stream to be revoked in the new audio and video forwarding relationship, the corresponding media forwarding channel is maintained before the common effective switching boundary is reached, and the old media termination boundary corresponding to the media stream to be revoked is determined according to the common effective switching boundary.

[0030] The collaborative migration module is used to stop the output of the media stream to be cancelled at the corresponding old media termination boundary when the common effective switching boundary is reached, and to make the new media stream start outputting pre-buffered media data from the corresponding effective media boundary.

[0031] Furthermore, the court hearing terminal includes at least two of the following: judge's terminal, court clerk's terminal, party's terminal, witness's terminal, and court hearing display terminal;

[0032] The forwarding relationship determination module is used to establish new audio and video forwarding relationships based on the court roles and media receiving permissions corresponding to different court terminals, so that different court terminals corresponding to the same court status change event have mutually distinct target media source combinations.

[0033] The state transition transaction management module is used to associate the new audio and video forwarding relationships corresponding to different court hearing terminals with the same state transition transaction.

[0034] Furthermore, the forwarding relationship determination module is also used to manage the media recording relationship corresponding to the court hearing recording equipment and the audio and video forwarding relationship corresponding to each court hearing terminal separately, and set the media recording relationship as a continuous forwarding relationship that does not change with the state transition transaction;

[0035] During the migration of the multiple court hearing terminals from the old audio and video forwarding relationship to the new audio and video forwarding relationship, the court hearing recording and broadcasting equipment maintains continuous reception and recording of the preset court hearing media stream according to the continuous forwarding relationship.

[0036] Furthermore, it also includes a state consistency verification module;

[0037] The state consistency verification module is used to obtain the currently effective state version identifier of each court terminal after the state migration transaction is submitted, and compare it with the state version identifier corresponding to the state migration transaction, and identify the court terminal with inconsistent state version identifier as an abnormal migration terminal.

[0038] The state transition transaction management module is used to determine whether the abnormal migration terminal is a critical migration terminal based on the court hearing role corresponding to the abnormal migration terminal and the current court hearing status.

[0039] When the abnormal migration terminal is a critical migration terminal, the court trial terminal associated with the state migration transaction that has completed migration is restored to the old audio and video forwarding relationship, and the switching boundary determination module redetermines the common effective switching boundary.

[0040] When the abnormal migration terminal is not a critical migration terminal, the new audio and video forwarding relationship corresponding to other court hearing terminals is maintained, and the media status detection module re-acquires the encoded frame status, buffer progress, and transmission progress of the media stream to be switched corresponding to the abnormal migration terminal. The switching boundary determination module determines the abnormal migration switching boundary based on the re-acquired encoded frame status, buffer progress, and transmission progress. The collaborative migration module re-executes the audio and video forwarding relationship migration of the abnormal migration terminal based on the abnormal migration switching boundary.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] This invention determines the old and new audio / video forwarding relationships and media streams to be switched among multiple courtroom terminals based on changes in courtroom status. It also determines candidate switching boundaries, switching reference times, and corresponding effective media boundaries by combining the encoded frame status, buffer progress, and transmission progress of each media stream to be switched. This allows multiple media streams with different encoded boundaries and transmission states to form a common effective switching boundary. By pre-establishing a new media forwarding channel before this boundary is reached and maintaining the validity of the old audio / video forwarding relationships, it avoids prematurely interrupting the old media stream before the new media stream meets the decoding and output conditions. Furthermore, it unifies the new audio / video forwarding relationships among multiple courtroom terminals through state transition transactions and collaboratively completes the forwarding relationship migration when the common effective switching boundary is reached. This reduces asynchronous switching between different terminals, residual old and new media, and audio / video misalignment during courtroom status changes, thereby improving the continuity and consistency of multi-terminal audio / video fusion forwarding in the court. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of the court audio and video multi-terminal fusion forwarding system of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the principle of determining the effective switching boundary of multiple media in this invention;

[0045] Figure 3 This is a schematic diagram illustrating the multi-terminal state migration transaction collaborative switching of the present invention;

[0046] Figure 4 This is a schematic diagram illustrating the switching and connection between the newly added media stream and the media stream to be cancelled in this invention;

[0047] Figure 5 This is a schematic diagram of the state consistency verification and anomaly migration handling of the present invention. Detailed Implementation

[0048] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are used to illustrate the technical principles and specific implementation processes of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adaptive adjustments to the specific implementation forms according to the type of court hearing business, the number of court hearing terminals, the media encoding method, the media forwarding protocol, and the configuration of court hearing role permissions without departing from the technical concept of the present invention.

[0049] This invention addresses the problem that during court hearings, changes in witness entry / exit, alterations in the subjects of evidence presentation and cross-examination, changes in speaking permissions, or changes in the hearing stage can simultaneously alter the multi-channel audio and video forwarding relationships of multiple court hearing terminals. Different media streams have independent encoded frame boundaries, buffer progress, and transmission progress, making it difficult for different court hearing terminals to coordinate and switch forwarding relationships under the same hearing state change. The invention determines the media streams to be switched based on the audio and video forwarding relationships before and after the hearing state change, determines candidate switching boundaries based on the encoded frame states of each media stream to be switched, and determines the switching reference time and the effective media boundaries corresponding to each media stream to be switched by combining buffer progress and transmission progress. When all media streams to be switched meet the switching conditions, the switching reference time is determined as the common effective switching boundary. A new media forwarding channel is pre-established before the common effective switching boundary is reached, while maintaining the validity of the old audio and video forwarding relationships. Through state transition transactions, multiple court hearing terminals collaboratively migrate to the new audio and video forwarding relationships after meeting the switching conditions.

[0050] Example 1

[0051] like Figure 1As shown, this embodiment provides a court audio and video multi-terminal fusion forwarding system, including a court hearing status management module, a forwarding relationship determination module, a media status detection module, a switching boundary determination module, a status migration transaction management module, a forwarding pre-establishment module, and a collaborative migration module.

[0052] The court's audio and video multi-terminal fusion forwarding system connects to multiple media source devices, including judge cameras, litigant cameras, witness cameras, courtroom microphones, witness microphones, and other devices used to collect audio and video information from court hearings. Each media source device generates a corresponding audio or video media stream and connects it to the court's audio and video multi-terminal fusion forwarding system. The system then forwards the media stream to different courtroom terminals according to the current hearing status, the parties involved, and media receiving permissions.

[0053] The court hearing terminals include at least two of the following: judge terminals, court clerk terminals, party terminals, witness terminals, and court hearing display terminals. Different court hearing terminals have different court hearing roles and media receiving permissions; therefore, under the same court hearing conditions, the target media source combinations corresponding to different court hearing terminals can be different.

[0054] For example, during witness testimony, the judge's terminal can receive witness video, witness audio, and media streams from the parties involved; the court clerk's terminal can receive preset media streams used for court recording; the parties' terminals can receive judge's media stream and witness's media stream; and the witness's terminal can receive judge's media stream and other authorized court media streams. When the court proceedings change from witness testimony to cross-examination after the witness has left the courtroom, the witness's media stream can be removed from the target media source combination of some court terminals, while the plaintiff's media stream, the defendant's media stream, or other media streams corresponding to the current cross-examination stage can be added to the target media source combination of the corresponding court terminals.

[0055] The court hearing status management module is used to acquire court hearing status change events and determine multiple court hearing terminals participating in the current status transition based on these events. These court hearing status change events include at least one of the following: witness entry, witness exit, change of evidence presenter, change of cross-examination subject, change of speaking authority, and change of court hearing stage. Court hearing status change events can be generated by court hearing control operations on the clerk's or judge's terminal, or they can be generated by the court hearing business system based on the current court hearing process status.

[0056] The forwarding relationship determination module determines the old audio and video forwarding relationship of each court hearing terminal before the state change and the new audio and video forwarding relationship after the state change based on the court hearing status change event. For any court hearing terminal Represent its old audio and video forwarding relationship as Represent its new audio and video forwarding relationship as .in, Used to distinguish the different court terminals participating in this status transition.

[0057] The audio and video forwarding relationship includes the target media source, media type and receiving permissions, as well as the media stream identifier, forwarding channel identifier, target terminal identifier, encoding type and the association between audio media stream and video media stream.

[0058] The forwarding relationship determination module determines the forwarding relationship of old audio and video files corresponding to the same court hearing terminal. Relationship with new audio and video forwarding The system compares media streams and determines the new, maintained, and revoked media streams based on the comparison results. The new media streams are those existing in the new audio / video forwarding relationship but not in the old one. The revoked media streams are those existing in the old audio / video forwarding relationship but no longer in the new one. The maintained media streams are those that need to continue being forwarded to the corresponding courtroom terminal before and after the state change. Therefore, the system does not need to re-establish all media forwarding relationships each time the courtroom state changes; instead, it can determine the media streams actually participating in the current state transition based on the differences between the old and new audio / video forwarding relationships.

[0059] The media state detection module acquires the encoded frame state of each media stream to be switched, and determines the candidate switching boundary of the corresponding media stream based on the encoded frame state. For video media streams, the encoded frame state includes the position of independently decodeable frames and their corresponding timestamps; for audio media streams, the encoded frame state includes audio frame boundaries and their corresponding timestamps.

[0060] The handover boundary determination module determines the handover reference time based on the candidate handover boundary, buffer progress, and transmission progress corresponding to each media stream to be switched, and determines the effective media boundary corresponding to the handover reference time for each media stream to be switched. When each media stream to be switched can complete the forwarding relationship switch based on the corresponding effective media boundary, the handover boundary determination module determines the handover reference time as the common effective handover boundary.

[0061] The State Transition Transaction Management module establishes a State Transition Transaction for the current trial state change event, and associates multiple trial terminals participating in this state transition and their corresponding new audio and video forwarding relationships with the same State Transition Transaction, so that the changes in the forwarding relationships of multiple terminals caused by a single trial state change can be managed according to a unified transition state.

[0062] Before the common effective switching boundary is reached, the forwarding pre-establishment module pre-establishes corresponding new media forwarding channels based on the new audio and video forwarding relationships of each court terminal, and prepares the new media data to be output after the state migration. During the establishment of new media forwarding channels and the preparation of new media data, the old audio and video forwarding relationships remain valid to avoid premature interruption of the current court media output due to the incomplete preparation of new media forwarding channels.

[0063] The collaborative migration module submits the state migration transaction when the media forwarding channels corresponding to each court terminal participating in the state migration meet the switching conditions and the common effective switching boundary is reached, so that the new audio and video forwarding relationship corresponding to each court terminal takes effect.

[0064] In one specific implementation, the court's audio and video multi-terminal fusion forwarding system can also be configured with signaling control service, time synchronization service, cache management service, transmission scheduling service, security authentication service, and log auditing service to support the operation of the aforementioned modules. The signaling control service transmits control information related to changes in court proceedings status, the establishment of media forwarding channels, and state transitions; the time synchronization service provides a unified time reference for comparing timestamps between different media streams; the cache management service manages the cache location and cache progress of media data; the transmission scheduling service manages the transmission process of media streams and obtains the corresponding transmission progress; the security authentication service verifies media forwarding relationships based on court roles and media receiving permissions; and the log auditing service records changes in court proceedings status, changes in media forwarding relationships, and state transition results. These services can be configured separately or integrated into one or more media servers depending on the actual deployment method.

[0065] The system can also connect to court hearing recording equipment, which includes a court hearing recording server and a recording and storage unit. The media recording relationship corresponding to the court hearing recording equipment and the audio and video forwarding relationship corresponding to each court hearing terminal are managed separately, and the media recording relationship can be set as a continuous forwarding relationship that does not change with the status migration and transaction switching of ordinary court hearing terminals.

[0066] For example, when a witness leaves the courtroom, the witness media stream can be withdrawn from the new audio-visual forwarding relationship corresponding to the judge's terminal, the party's terminal, or the court hearing display terminal. However, for witness media data that needs to be saved according to the court hearing recording rules, the court hearing recording server still receives the corresponding media stream according to the preset media recording relationship and sends the received media data to the recording storage unit. Thus, each court hearing terminal can change the media content presented in real time according to the court hearing status, while the court hearing recording equipment can still maintain the continuous reception and recording of court hearing media data according to the preset recording rules.

[0067] Example 2

[0068] like Figure 2As shown, this embodiment uses the example of terminals A, B, and C participating in the same court hearing state transition to illustrate the determination of the common effective switching boundary and the collaborative transition process of multiple court hearing terminals. Terminal A corresponds to the interconnected video streams. and audio stream Terminal B corresponds to the interconnected video streams. and audio stream Terminal C corresponds to the video stream The media streams are used to illustrate the common switching process under different encoding boundaries and different transmission states, and are not used to limit the number of court terminals and media streams actually involved in the state transition.

[0069] For video media streams Figure 2 In the video coding sequence, I-frames ①, ②, and ③ represent independently decodeable frames at different positions, while P-frames represent predictive coded frames that require a reference frame for decoding. When a new video media stream needs to be output to the courtroom terminal after a state transition, if the output starts directly from a P-frame that depends on a preceding reference frame, the courtroom terminal may not be able to immediately complete normal video decoding. Therefore, the media state detection module detects independently decodeable frames in the video media stream and determines the positions of independently decodeable frames that can serve as the starting point for decoding a new video media stream as video candidate switching boundaries.

[0070] For audio media streams Figure 2 In this context, frame A represents an audio frame. The media state detection module obtains the audio frame boundary and corresponding timestamp of each audio frame, and determines the audio candidate switching boundary based on the timestamp relationship between the audio frame boundary and the associated video media stream.

[0071] When a new audio and video forwarding relationship corresponding to the same court hearing terminal includes both related video and audio media streams, the media state detection module further determines the terminal-level candidate switching boundary corresponding to that court hearing terminal based on the video candidate switching boundary, the audio candidate switching boundary, and a preset synchronization tolerance. The preset synchronization tolerance is used to limit the allowable time deviation between related audio and video media streams. Therefore, the video candidate switching boundary and the audio candidate switching boundary do not need to have exactly the same physical frame position; as long as the difference between their corresponding timestamps is within the preset synchronization tolerance range, they can be considered as a set of terminal-level candidate switching boundaries corresponding to the same court hearing terminal.

[0072] For terminal C, which only involves video media streams, it can be based on the video stream. The independently decodeable frames directly determine the corresponding terminal-level candidate switching boundaries. For other implementations involving only audio media streams, the corresponding terminal-level candidate switching boundaries can also be determined based on the boundaries of audio frames that meet the output conditions. Each media stream to be switched can have one or more candidate switching boundaries. The switching boundary determination module determines the candidate switching time interval for selecting the switching reference time based on the terminal-level candidate switching boundaries corresponding to each court terminal participating in this state transition.

[0073] The start time of the candidate switching time interval is expressed as follows: The end time is represented as The candidate switching time interval This is used to define the time range within which a switching reference time can be selected for this state transition. The candidate switching time interval is determined jointly by multiple terminal-level candidate switching boundaries participating in this state transition, rather than being determined solely based on the candidate switching boundary of one media stream.

[0074] Because the encoding cycle, network transmission latency, current buffer size, and media output progress of each media stream may differ, even if candidate switching boundaries have appeared for each media stream to be switched, it cannot be directly determined whether multiple courtroom terminals can immediately transition to a new state. Therefore, the switching boundary determination module further acquires the buffering progress and transmission progress of each media stream to be switched within the candidate switching time interval. The buffering progress and transmission progress, progressing over time, indicate the extent to which the corresponding media data has entered the buffer or completed transmission to the target processing location. The remaining processing time indicates the processing time required for the currently buffered or transmitted old media data to be fully output.

[0075] For any media stream to be switched The remaining processing time is expressed as The remaining processing time The remaining processing time is determined based on at least one of the following: the amount of old media data that has not yet been output, the media output rate, the amount of data that has not yet been consumed in the buffer, and the amount of data that has not yet been transmitted in the transmission queue. For example, the remaining processing time can be determined based on the duration of the media data that has not yet been output in the current buffer queue, or it can be adjusted by combining the amount of data to be transmitted in the transmission queue.

[0076] The switching boundary determination module determines the earliest time that each media stream is allowed to enter the new forwarding relationship after completing the output of the old media data, based on the remaining processing time of each media stream to be switched, and within the candidate switching time interval. Candidate times are detected sequentially. If a candidate time is not earlier than the earliest allowed switching time for each media stream to be switched, and each media stream to be switched can be located at a media boundary that satisfies the conditions of independent decoding or synchronous output at that candidate time, then the earliest candidate time that satisfies the above conditions is determined as the switching reference time. Therefore, the reference time is switched. The time point is not arbitrarily selected, but rather the earliest executable collaborative handover time that simultaneously satisfies the remaining processing status and media boundary conditions of each media stream to be switched within the candidate handover time interval; the handover reference time. Once determined, the switching boundary determination module determines the effective media boundary corresponding to the switching reference time for each media stream to be switched.

[0077] Specifically, video stream Audio stream Video stream Audio stream and video stream The corresponding effective media boundaries can be represented as follows: , , , and Each valid media boundary represents the media position in which the corresponding media stream is actually used to complete the output of old media data or the output of new media data during this state transition. Valid media boundaries corresponding to different media streams are not required to have exactly the same physical location in their respective media frame sequences, nor are their timestamps required to be exactly equal. Instead, they are determined by their respective timestamps relative to the switching reference time. Establish a corresponding relationship and meet the decoding and synchronization conditions of the corresponding media streams.

[0078] For example, video stream Corresponding effective media boundaries It can be determined at the time of switching the reference. A nearby independently decodeable frame location, audio stream Corresponding effective media boundaries The boundaries of audio frames whose timestamp differences with the independently decodeable frames are within a preset synchronization tolerance range can be determined. (Video stream) Audio stream and video stream The corresponding valid media boundaries can be determined according to the same principle. This applies when each media stream to be switched has a connection to the switching reference time. When the corresponding valid media boundary is met, and the media stream requiring independent decoding can be decoded normally from the corresponding valid media boundary, the interconnected audio and video media streams meet the preset synchronization tolerance, and the buffering and transmission progress of each media stream can meet the continuous output requirements after state transition, the switching boundary determination module will determine the switching reference time. It was determined to be a common effective handover boundary.

[0079] Therefore, the common effective handover boundary in this embodiment does not require all media streams to be switched to perform handover at exactly the same physical frame position, but rather uses the handover reference time. It serves as a unified time coordination benchmark for state transitions among multiple court hearing terminals, and utilizes the effective media boundaries corresponding to each media stream to complete the connection of specific media data.

[0080] Furthermore, such as Figure 3 As shown, once the common effective switching boundary is determined, the state transition transaction management module establishes a state transition transaction for the current court hearing state change, representing this state transition transaction as follows: and for state transition transactions Configuration status version identifier .in, Used to identify a specific multi-terminal state transition transaction. The two are used to identify the new audio / video forwarding relationship version corresponding to the state transition transaction, and they represent different technical objects.

[0081] State transition transactions The system unifies and links the multiple courtroom terminals involved in this state migration, the new audio and video forwarding relationships for each terminal, and the state version identifier. Judge terminals, court clerk terminals, litigant terminals, witness terminals, and court hearing display terminals can all be associated with the same state transition transaction.

[0082] For the judge's terminal, its old audio / video forwarding relationship and new audio / video forwarding relationship can be represented as follows: and For the clerk's terminal, the old and new audio / video forwarding relationships can be represented as follows: and Other courtroom terminals can represent the audio and video forwarding relationship before and after their status changes in the same way.

[0083] Before the state migration transaction is formally submitted, the forwarding pre-establishment module pre-establishes the corresponding new media forwarding channels according to the new audio and video forwarding relationships of each court hearing terminal, and keeps the old audio and video forwarding relationships valid during the process of pre-establishing the new media forwarding channels.

[0084] Once the new media forwarding channel is established, the forwarding pre-establishment module receives and caches the corresponding new media data. When the new media forwarding channel corresponding to a certain court hearing terminal is established and the new media stream corresponding to that terminal reaches the preset caching conditions, the state transition transaction management module or the forwarding pre-establishment module marks the court hearing terminal as pending switching.

[0085] The preset caching conditions can be set according to the media type. For example, for video media streams, it can be required that at least one start frame that can be independently decoded and a preset number of video frames following it have been cached; for audio media streams associated with video media streams, it can be required that audio frames that meet the preset synchronization tolerance with the corresponding video media stream have been cached; or it can be determined whether the new media stream meets the preset caching conditions based on whether the duration corresponding to the cached data reaches the preset caching duration.

[0086] When state transition transactions All associated court hearing terminals have entered the handover standby state and are working together to effectively switch boundaries. Upon arrival, the collaborative migration module submits the state transition transaction. This ensures that the new audio and video forwarding relationships corresponding to each court hearing terminal are based on the status version identifier. Effective.

[0087] Therefore, the multiple court terminals involved in the same court hearing status change do not switch separately according to the arrival time of their respective control instructions, but rather perform a coordinated state migration according to a common and effective switching boundary after the new media forwarding channels and media data are prepared separately.

[0088] The procedure for handling situations where the current shared valid handover boundary fails. This applies when at least one courtroom terminal fails to reach the current shared valid handover boundary. If the state has not yet entered the transition state before arrival, the state transition transaction management module will establish the current common valid transition boundary. Marked as invalid, the state migration transaction is not forced to be committed according to the invalidation boundary.

[0089] For courtroom terminals that are not yet ready, the media status detection module reacquires the encoded frame status, buffer progress, and transmission progress of their corresponding media stream after the current failure boundary, and redetermines the terminal-level candidate switching boundary after the current failure boundary. For other courtroom terminals that are already in the waiting-to-switch state, the system similarly determines their subsequent terminal-level candidate switching boundaries after the current failure boundary.

[0090] The handover boundary determination module re-determines the common effective handover boundary based on the subsequent terminal-level candidate handover boundaries of each court terminal after the current failure boundary, and represents the re-determined common effective handover boundary as follows: ; at the new common effective handover boundary Before arrival, the courtroom terminals that have already been prepared will maintain their old audio and video forwarding relationships and continue to maintain the established new media forwarding channels. The system will then proceed according to the new common and valid switching boundaries after all courtroom terminals have re-met the collaborative switching conditions. Commit the state transition transaction.

[0091] Furthermore, such as Figure 4 As shown in the figure, this embodiment also explains the specific connection method between the newly added media stream and the media stream to be withdrawn in the new audio and video forwarding relationship.

[0092] Figure 4 China-Israel video stream and audio stream This indicates older media streams that are to be withdrawn, in the form of video streams. and audio stream This indicates the need to add new media streams to the new audio / video forwarding relationship; the video stream will remain in place until the common effective switching boundary is reached. and audio stream The audio and video transmission continues to be routed to the corresponding courtroom terminal according to the old audio and video forwarding relationship. The system determines the old media termination boundary based on the common effective switching boundary and the media frame state of the old media stream, and represents the old media termination boundary as... The old media termination boundary is used to define the media position at which the media stream to be revoked stops outputting to the corresponding court hearing terminal. For video media streams, the corresponding termination position can be determined based on the output of complete video frames near the common effective switching boundary; for audio media streams, the stop output position can be determined based on the corresponding audio frame boundary.

[0093] For the new video stream and audio stream The forwarding pre-establishment module pre-establishes the corresponding media forwarding channel before the common effective handover boundary arrives. After the new media forwarding channel is established, it first enters the waiting reception state, and then pre-buffers the media data that meets the decoding and synchronization requirements upon receipt.

[0094] Figure 4 In This indicates the valid media boundaries corresponding to the newly added media stream. For video streams... The effective media boundary can correspond to the I-frame position that can be independently decoded; for audio streams The corresponding effective media boundary can be determined based on the time relationship between the audio frame timestamp and the effective media boundary of the associated video media stream.

[0095] The forwarding pre-establishment module can cache data for newly added media streams starting from a preset starting position before the corresponding valid media boundary, ensuring that the system has new media data that meets the requirements for continuous output before the common valid switching boundary is reached. The preset starting position can be determined based on the media encoding structure, network transmission status, and preset buffer duration.

[0096] Effective Media Boundaries Jointly and effectively switch boundaries and the termination boundary of old media These represent different media processing locations. Among them, the effective media boundary... Used to determine the media location where a new media stream meets the conditions for independent decoding and synchronous output, thus effectively switching boundaries. Used as a unified time coordination boundary for the transition of execution status across multiple court hearing terminals; old media termination boundary. Used to determine the position where the output of the media stream to be canceled stops.

[0097] Therefore, the three boundaries do not need to be represented by the same physical frame position on the media timeline. The system pre-establishes new media forwarding channels and performs pre-buffering before the common effective handover boundary arrives, ensuring that new media streams are ready for output while simultaneously maintaining the valid output of media streams awaiting cancellation. When the common effective handover boundary arrives... Upon arrival, the collaborative migration module causes the media stream to be cancelled to stop outputting according to the corresponding old media termination boundary, and causes the new media stream to output pre-buffered media data starting from the corresponding valid media boundary according to the new audio and video forwarding relationship.

[0098] For example, when the court proceedings switch from witness testimony to cross-examination after the witness has left the courtroom, the system can identify the witness video and audio streams as media streams to be withdrawn for some court terminals, and identify the corresponding parties' video and audio streams as new media streams. The system first establishes a forwarding channel for the new party media streams and pre-caches the media data, while maintaining the continued output of the witness media streams. When all court terminals reach the switchable state and the common effective switch boundary is reached, the system then migrates the forwarding relationship according to the effective media boundary and the old media termination boundary corresponding to each media stream. This avoids the discrepancy where some terminals are already displaying the new cross-examination screen while others remain on the witness testimony screen.

[0099] Example 3

[0100] like Figure 5As shown, this embodiment describes the state consistency verification and abnormal migration handling process after the state migration transaction is submitted; after the state migration transaction is submitted, the state consistency verification module obtains the current effective state version identifier of each court terminal participating in this state migration, and compares the current effective state version identifier of each court terminal with the state version identifier corresponding to this state migration transaction.

[0101] When the current state version identifier of a court hearing terminal matches the state version identifier corresponding to the state migration transaction, the state consistency verification module determines that the court hearing terminal has completed the state migration as expected. When the current state version identifier of a court hearing terminal does not match the state version identifier corresponding to the state migration transaction, the state consistency verification module determines that the court hearing terminal is an abnormal migration terminal.

[0102] The status version identifier can be stored in the currently effective media relationship configuration of each court hearing terminal, or it can be maintained by the media forwarding server based on the audio and video forwarding relationship currently used by each terminal. The status consistency verification module can determine the currently effective status version identifier of each court hearing terminal through terminal feedback information, effective configuration records in the media server, or a combination of both.

[0103] When no abnormal migration terminals are found, the state migration transaction management module confirms that all courtroom terminals participating in this state migration have migrated to the corresponding new audio / video forwarding relationship according to the state version identifier, and updates the current forwarding relationship status of the system. Afterwards, each courtroom terminal continues to receive the courtroom media stream according to the new audio / video forwarding relationship, and this state migration transaction is completed.

[0104] When an abnormal migration terminal exists, the state migration transaction management module further determines whether the abnormal migration terminal is a critical migration terminal based on the court hearing role corresponding to the abnormal migration terminal and the current court hearing status.

[0105] The critical migration terminal refers to a court terminal whose failure to migrate its state would affect the consistency of the multiple terminal states required for the current trial stage or the presentation of necessary trial media. The critical migration terminal can be dynamically determined based on the trial role and the current trial status, rather than being permanently fixed according to the terminal type.

[0106] In one specific implementation, the system pre-establishes a correspondence between court hearing status, court hearing roles, and key migration attributes. When the court hearing role corresponding to an abnormal migration terminal is configured to participate in necessary media interaction, speech control, or court hearing status consistency control in the current court hearing status, the abnormal migration terminal is identified as a key migration terminal; when its state migration failure does not affect the media interaction and status consistency between other necessary court hearing terminals in the current court hearing stage, it is identified as a non-key migration terminal.

[0107] For example, during the witness testimony phase, the judge's terminal and the witness's terminal can be designated as key migration terminals based on the current trial requirements; after the witness has left the court and the subsequent cross-examination phase has begun, the witness's terminal may no longer be designated as a key migration terminal for that trial phase. As another example, when the court clerk's terminal performs the necessary trial record confirmation function for the current trial phase, it can be designated as a key migration terminal.

[0108] When an abnormally migrated terminal is a critical migration terminal, the state migration transaction management module restores the relevant court hearing terminals that have completed migration in this state migration transaction to the old audio and video forwarding relationship, so as to avoid multiple critical court hearing terminals being in different court hearing media states for a long time.

[0109] In one specific implementation, if the old media forwarding channel remains active within a preset retention time after the common effective switching boundary is reached, the old audio and video forwarding relationship can be directly restored using the retained old media forwarding channel. If the old media forwarding channel has been released, the corresponding media forwarding channel can be re-established based on the old audio and video forwarding relationship saved in the state transition transaction. Before committing, the state transition transaction can save the old audio and video forwarding relationship corresponding to each court terminal participating in the migration for use in case of abnormal recovery.

[0110] After the relevant court hearing terminals are restored to the old audio and video forwarding relationship, the media state detection module re-acquires the encoded frame state corresponding to the media streams participating in the subsequent migration. The handover boundary determination module re-determines the candidate handover boundary based on the encoded frame state, and re-determines the handover reference time and common effective handover boundary in combination with the current buffer progress and transmission progress. Once the relevant court hearing terminals meet the handover conditions again, the corresponding state migration transaction is submitted again.

[0111] When the abnormally migrated terminal is not a critical migration terminal, the state migration transaction management module keeps the new audio and video forwarding relationship of other successfully migrated court hearing terminals valid, and does not require other court hearing terminals to be restored to the old audio and video forwarding relationship as a whole.

[0112] The media status detection module re-acquires the encoded frame status corresponding to the media stream to be switched for the abnormally migrating terminal, and determines new candidate switching conditions based on the current buffering progress and transmission progress of the media stream to be switched. The switching boundary determination module redetermines the abnormal migration switching boundary based on the encoded frame status, buffering progress, and transmission progress of the media stream corresponding to the abnormally migrating terminal.

[0113] Once the new media forwarding channel corresponding to the abnormal migration terminal meets the establishment conditions and preset caching conditions again, the collaborative migration module re-executes the audio and video forwarding relationship migration of the abnormal migration terminal according to the abnormal migration switching boundary, so that the abnormal migration terminal finally enters the new audio and video forwarding relationship corresponding to the current court hearing status.

[0114] Therefore, when a state migration transaction fails, this embodiment does not uniformly roll back all terminals experiencing the failure. Instead, it assesses the impact of the terminal's role in the court proceedings and its current state on the consistency of state across multiple terminals. When a critical migration terminal fails, the old audio and video forwarding relationships of the relevant court proceedings terminals are restored, and the common effective switching boundary is redefined, allowing the critical court proceedings terminals to re-execute the collaborative migration. When a non-critical migration terminal fails, the new audio and video forwarding relationships already in effect for other court proceedings terminals are maintained, and only the terminal experiencing the failure undergoes a partial compensation migration. This approach maintains the necessary consistency of court proceedings state while minimizing the impact of the failure migration on the normal media output of other court proceedings terminals.

[0115] Through the above implementation methods Figure 1 The system modules and multi-terminal media forwarding relationships shown are as follows: Figure 2 The relationship between the candidate handover boundary and the common effective handover boundary is shown. Figure 3 The state transition transactions and common valid switch boundary failure recalculation relationships shown are as follows: Figure 4 The connection between newly added media streams and media streams to be withdrawn is shown. Figure 5 The state consistency verification and anomaly migration handling relationships shown form a complete technical chain. When a change in the state of a court hearing simultaneously causes changes in the multi-channel audio and video forwarding relationships of multiple court hearing terminals, the system can determine the actual switchable media position of each media stream based on its own encoded frame boundary, buffer progress, and transmission progress. It also coordinates the state migration of multiple court hearing terminals with a common and effective switching boundary, thereby reducing inconsistencies in the court hearing states of different terminals, residual old and new media streams, and misalignment of associated audio and video caused by asynchronous modification of forwarding relationships on a terminal-by-terminal basis.

[0116] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A court audio and video multi-terminal fusion and forwarding system, characterized in that, It includes a court hearing status management module, a forwarding relationship determination module, a media status detection module, a handover boundary determination module, a status migration transaction management module, a forwarding pre-establishment module, and a collaborative migration module; The court hearing status management module is used to determine multiple court hearing terminals participating in the status transition based on court hearing status change events; The forwarding relationship determination module is used to determine the old and new audio and video forwarding relationships before and after the status change of each court terminal, and thereby determine the media stream to be switched. The media state detection module is used to determine the candidate switching boundary based on the encoded frame state of each media stream to be switched. The switching boundary determination module is used to determine the switching reference time based on the candidate switching boundary, buffer progress and transmission progress, and to determine the effective media boundary corresponding to each media stream to be switched and the switching reference time. When each media stream to be switched meets the condition of switching based on the corresponding effective media boundary, the switching reference time is determined as the common effective switching boundary. The state transition transaction management module is used to establish state transition transactions and associate the multiple court hearing terminals and their new audio and video forwarding relationships with the state transition transactions. The forwarding pre-establishment module is used to pre-establish the media forwarding channel corresponding to the new audio and video forwarding relationship before the common effective switching boundary is reached, and to keep the old audio and video forwarding relationship effective; The collaborative migration module is used to submit the state migration transaction when the media forwarding channel meets the switching conditions and the common effective switching boundary is reached, so that the multiple court hearing terminals can collaboratively migrate to the new audio and video forwarding relationship.

2. The court audio and video multi-terminal fusion forwarding system according to claim 1, characterized in that, The events that change the status of the trial include at least one of the following: witness entry or exit from the court, change of the subject of evidence presentation and cross-examination, change of speaking authority, and change of trial stage. The forwarding relationship determination module is used to determine the target media source, media type and receiving permissions corresponding to each court hearing terminal based on the changed court hearing status, and compare it with the terminal media relationship configuration before the change to determine the added, kept or withdrawn media streams.

3. The court audio and video multi-terminal fusion and forwarding system according to claim 1, characterized in that, The media state detection module is used to determine the video candidate switching boundary based on the independently decodeable frames of the video media stream to be switched, and to determine the audio candidate switching boundary based on the audio frame boundary of the audio media stream to be switched and its timestamp relationship with the corresponding video media stream. Based on the interrelated video candidate switching boundaries, audio candidate switching boundaries, and preset synchronization tolerances, the terminal-level candidate switching boundaries of the corresponding court hearing terminal are determined.

4. The court audio and video multi-terminal fusion forwarding system according to claim 3, characterized in that, The switching boundary determination module is used to determine the candidate switching time interval based on the candidate switching boundary of each terminal, determine the remaining processing time required to output the current old media data based on the buffer progress and transmission progress of each media stream to be switched, and determine the switching reference time from the candidate switching time interval accordingly. For each media stream to be switched, a valid media boundary is determined that corresponds to the switching reference time and satisfies the conditions of independent decoding and synchronous output.

5. The court audio and video multi-terminal fusion forwarding system according to claim 1, characterized in that, The state transition transaction management module is used to configure a state version identifier for the state transition transaction and associate it with multiple court hearing terminals and the corresponding new audio and video forwarding relationships. The forwarding pre-establishment module is used to mark the corresponding court hearing terminal as pending switching state after the new media forwarding channel is established and the new media stream reaches the preset caching conditions; The collaborative migration module is used to ensure that, after all court terminals are in a state of waiting to be switched, the new audio and video forwarding relationships take effect at the common effective switching boundary according to the state version identifier.

6. The court audio and video multi-terminal fusion forwarding system according to claim 5, characterized in that, When a court hearing terminal fails to enter the waiting state before the current common valid handover boundary is reached, the state transition transaction management module marks the current common valid handover boundary as invalid. The switching boundary determination module re-determines the terminal-level candidate switching boundary based on the encoded frame status, buffering, and transmission progress of the media stream corresponding to the unready terminal after the current failure boundary, and determines the subsequent terminal-level candidate switching boundaries of other court hearing terminals after the current failure boundary, and re-determines the common effective switching boundary based on each subsequent terminal-level candidate switching boundary.

7. The court audio and video multi-terminal fusion forwarding system according to claim 1, characterized in that, The forwarding pre-establishment module is used to establish a media forwarding channel for the new media stream before the common effective switching boundary is reached, and to cache media data starting from a preset starting position before the corresponding effective media boundary, maintain the media forwarding channel corresponding to the media stream to be cancelled, and determine its old media termination boundary. The collaborative migration module is used to stop the output of the media stream to be canceled at the old media termination boundary when the common effective switching boundary is reached, and to make the new media stream start outputting pre-buffered media data from the corresponding effective media boundary.

8. The court audio and video multi-terminal fusion and forwarding system according to claim 1, characterized in that, The court hearing terminal includes at least two of the following: judge's terminal, court clerk's terminal, party's terminal, witness's terminal, and court hearing display terminal; The forwarding relationship determination module is used to establish new audio and video forwarding relationships with different target media source combinations based on the court hearing roles and media receiving permissions of each court hearing terminal. The state transition transaction management module is used to uniformly associate the new audio and video forwarding relationships of different court hearing terminals with the same state transition transaction.

9. The court audio and video multi-terminal fusion and forwarding system according to claim 8, characterized in that, The forwarding relationship determination module is used to manage the media recording relationship corresponding to the court hearing recording equipment and the audio and video forwarding relationship of each court hearing terminal respectively, and set the media recording relationship as a continuous forwarding relationship that does not change with state migration transactions; During the state transition of multiple court hearing terminals, the court hearing recording and broadcasting equipment maintains continuous reception and recording of the preset court hearing media stream according to the continuous forwarding relationship.

10. The court audio and video multi-terminal fusion forwarding system according to claim 5, characterized in that, It also includes a state consistency verification module, which is used to compare the actual effective state version identifier of each court terminal after state migration with the state version identifier corresponding to the state migration transaction to identify abnormal migration terminals. The state transition transaction management module is used to determine whether the abnormal migration terminal is a critical migration terminal based on its court hearing role and current court hearing status. If the terminal is a critical migration terminal, restore the migrated court hearing terminal to the old audio and video forwarding relationship and redetermine the common effective switching boundary; if the terminal is not a critical migration terminal, maintain the new audio and video forwarding relationship of other court hearing terminals and redetermine the abnormal migration switching boundary for the abnormal migration terminal.

Images