Streaming media data encoding, decoding and storing method for court hearing

Abstract

The invention relates to the field of real-time video data processing methods, in particular to a streaming media data encoding, decoding and storing method for court hearing, which comprises the following steps of: encoding streaming media data in different data formats according to original data processing, H.264 encoding, organization of an NALU structure and encapsulation format selection, and transmitting the encoded data; decoding the received coded data and then playing the decoded coded data; in the encoding process, low-delay transmission is carried out based on an RTP protocol of a UDP, the overhead of a protocol head is reduced according to a preset proportion, RTP data packets are optimally divided and packaged according to the characteristics of streaming media data, and then the sending rate of the RTP data packets is dynamically adjusted according to the network congestion condition and the feedback of a receiving end, so that the RTP data packets are rapidly transmitted according to a real-time priority strategy. According to the invention, the information interaction effect can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of real-time video data processing methods, and in particular to streaming media data encoding, decoding and storage methods used in court trials. Background Art

[0002] With the rapid development of information technology, the sharing of various information is being transmitted to a wider range, especially for a small number of the general public who have limited access to information, such as court-related information. In recent years, in order to let the public understand relevant legal information, the call for cloud-based shared courts has become increasingly louder. Cloud-based shared courts are a type of people's court that is built in digital space and virtualized around the masses. It relies on the existing hardware facilities of towns, streets, villages, communities, and industry organizations, and is deployed and spread across towns, streets, villages, communities, and industry organizations with the standard configuration of "one network cable and one screen", effectively solving the problem of the overall small number of people's courts and the large service radius, allowing judicial services to reach every place that the network can reach.

[0003] At present, the sharing of judicial services is generally carried out through real-time or recorded video broadcasting on television networks. Information transmission is delayed, and the public can only obtain information through limited television channels, which is not real-time and interactive. To this end, the cloud-based shared court is developing in the direction of diversified functions and diversified interactions, so that the cloud-based shared court is no longer a simple information sharing, and thus plays an important role in resolving conflicts and disputes at the grassroots level, forming an integrated closed loop of conflict and dispute mediation and resolution. It improves the efficiency and quality of judicial work through modern information technology, and realizes litigation services that transcend geographical restrictions, are convenient, fast, fair and efficient.

[0004] In order to realize the diversified interactive functions of the shared court on the cloud, the problem that the cloud court urgently needs to solve is the encoding and decoding of data interaction between multiple different information sources. To address this problem, the encoding and decoding of information data interaction requires the processing of audio, video, text and other information. However, given the differences in the source devices and information formats of the information sources involved in the cloud interaction, such as the data and equipment differences between judicial-related departments and external judicial departments, between the intranet and extranet of judicial departments, and between the judicial departments and the devices used by the parties, the encoding and decoding of information data is very important. If encoding and decoding are performed separately for each type of information data, it is easy to cause delays and freezes in information interaction, and in severe cases, even interrupt the interaction. Summary of the invention

[0005] The present invention aims to provide a streaming media data encoding and decoding method for court trials, so as to encode and decode information data from multiple sources, prevent interaction delays and freezes, and prevent interaction interruptions.

[0006] The streaming media data encoding and decoding method used in the trial in this scheme includes the following contents:

[0007] Encode streaming media data in different data formats according to original data processing, H.264 encoding, NALU structure organization and encapsulation format selection, and transmit the encoded data;

[0008] Decode the received encoded data and play it back;

[0009] During the encoding process, the UDP-based RTP protocol performs low-latency transmission and reduces the protocol header overhead according to a preset ratio. The RTP data packets are optimized for division and encapsulation according to the characteristics of the streaming media data. The sending rate of the RTP data packets is dynamically adjusted according to the network congestion conditions and feedback from the receiving end, so as to achieve fast transmission with a real-time priority strategy.

[0010] The beneficial effects of this program are:

[0011] The cloud court information encoding and decoding of this solution is adaptable to data processing of different formats. It can be processed on the basis of the original equipment, saving the equipment cost of the overall transformation. It can interact with data from different sources after encoding or decoding, so that the information interaction is minimally disturbed by encoding or decoding, preventing delays and freezes, and preventing interruptions in the interaction process.

[0012] Preferably, in order to avoid waiting for retransmission and reduce the delay of data transmission, the real-time priority strategy is to process the occasionally lost data packets through a preset error correction mechanism.

[0013] Preferably, in order to avoid unnecessary padding and segmentation and improve data transmission efficiency, when optimizing the division and encapsulation of RTP data packets, by comparing the lengths of each field such as timestamp, sequence number, load format, adapted network conditions and statistical information, the fields of preset length are divided to the front for optimization and division, and the meaningless or redundant information in each field is deleted before encapsulation.

[0014] Preferably, in order to recover the lost original packets, reduce screen freezes and audio interruptions, and make the playback effect as smooth as possible and reduce obvious distortion, the preset error correction mechanism is:

[0015] When sending, the original media data is encoded using the RS error correction coding algorithm;

[0016] It can conceal the data errors of packets lost during transmission and selectively retransmit the data loss of key frames or important audio segments detected;

[0017] The buffer size is adaptively adjusted based on whether the network condition is good or not.

[0018] Preferably, in order to provide clear and sharp images and display the details of the shared court on the cloud, it also includes, during the encoding process, automatically adjusting the picture parameters according to the playback environment and the playback status of the terminal device to enhance the picture quality display condition;

[0019] During the encoding and decoding process, multiple audio formats and channels can be set, and the audio equalization adjustment function is available.

[0020] Preferably, in order to maintain the integrity of interactive data recording and storage, the authenticity of the data, and the logical clarity of the previous and subsequent data, the following contents are also included:

[0021] The trial control operation is carried out with simple interface operation buttons, and the trial control operation is encrypted, automatically recovered and fault-tolerant;

[0022] The process text information obtained by real-time recording is synchronously obtained in chronological order, and the process text information is intelligently displayed to different source parties in combination with the external environment.

[0023] Preferably, in order to allow the source parties of different ages, different vision conditions, different comprehension speeds and different comprehension abilities to quickly and accurately access the key and critical information in the interaction process, the intelligent display assesses the cultural level of the parties according to the age, work and registered residence of the source party, and determines the central idea information in the interaction data based on the cultural level for prominent display; when the cultural level is lower than the preset requirement, the central idea information is translated into simple information that is easy to understand for prominent display.

[0024] Preferably, in order to make the intelligent display clearer and highlight the key points, the highlighted display is displayed through color identification, color and contrast adjustment, and font size.

[0025] The streaming media data storage method used for court trials includes the following:

[0026] The video data of the interactive process is stored in a preset area for operation management, and the storage location and backup strategy are automatically adjusted according to the importance and access frequency of the video using a preset algorithm;

[0027] Operate and feedback the video recording tasks, customize the parameters of the video recording tasks, intelligently analyze the video content, automatically identify important events and abnormal situations, and provide auxiliary decision-making for the video recording tasks;

[0028] Operational management of historical video recording tasks, quick retrieval and query, and automatic generation of video summaries of historical videos;

[0029] The compatible playback component automatically adjusts the video playback quality according to network conditions and device performance, and supports multi-screen playback.

[0030] Preferably, in order to facilitate operation, it also includes:

[0031] Perform video synthesis, decomposition, text overlay, image overlay and special effects operations;

[0032] Set up multiple API interfaces and perform security verification on the API interfaces.

[0033] Compared with the prior art, the beneficial effects of this solution are:

[0034] The streaming media data at the trial site is encoded and decoded to adapt to the different versions and different models of terminal data integration of different application users, so that on the basis of the existing platform, without adding additional equipment or updating new equipment, the trial site can accurately play data of different formats, and improve the fluency, timeliness, accuracy and clarity of information interaction among multiple parties at the trial site. At the same time, the data during the trial process is recorded in real time, and the corresponding trial data is intelligently displayed, so that party users with different vision, hearing, and cultural levels can accurately capture the key and important information in the trial interaction process, so as to better understand the situation and information during the trial process and ensure the smooth, orderly and rapid progress of the trial. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of Embodiment 1 of the method for encoding and decoding streaming media data for court trial of the present invention;

[0036] Figure 2 This is a flowchart of Embodiment 3 of the streaming media data encoding and decoding method for court trials of the present invention. DETAILED DESCRIPTION

[0037] The following is further explained in detail through specific implementation methods.

[0038] Embodiment 1

[0039] Streaming media data encoding and decoding methods used in court trials, such as Figure 1 As shown, including the following:

[0040] The streaming media data in different data formats are encoded according to the original data processing, H.264 encoding, NALU structure organization and encapsulation format selection. The specific encoding of the streaming media data is performed according to the data format of the target terminal to be transmitted, and the encoded data is transmitted, for example, the encoded data is transmitted to user terminals in different locations and different versions; the received encoded data is decoded and played, the data format is such as MP4, AVI, MKV, FLV, etc., and the decoding of the data is performed according to the specific data format of the received user terminal. The specific encoding and decoding principles and algorithms are existing and will not be repeated here.

[0041] During the encoding process, the UDP-based RTP protocol performs low-latency transmission, reduces the protocol header overhead according to a preset ratio, optimizes the division and encapsulation of RTP packets according to the characteristics of streaming data, and dynamically adjusts the sending rate of RTP packets according to network congestion conditions and feedback from the receiving end, so as to quickly transmit with a real-time priority strategy. The RTP protocol header is relatively small, which reduces the amount of extra data caused by header information in network transmission, thereby reducing transmission delay.

[0042] Dynamically adjusting the data packet sending rate mainly relies on the feedback mechanism provided by the RTCP (Real-time Transport Control Protocol) protocol. RTCP works with RTP (Real-time Transport Protocol) to provide end-to-end real-time transmission capabilities and provide feedback on the current network status and data reception quality. Through this feedback information, the server can control data transmission, thereby adjusting the data transmission rate according to changes in network status, achieving uninterrupted and smooth playback of streaming media services, improving user perception quality, and avoiding exacerbating network congestion.

[0043] The real-time priority strategy is to process the occasional lost data packets through the preset error correction mechanism. The preset error correction mechanism is:

[0044] When sending, the original media data is encoded using the RS error correction coding algorithm. For example, for every N original data packets, M redundant data packets are generated. When some data packets are lost, the receiving end calculates these redundant data packets and the received valid data packets to recover the lost information. Assuming that 2 redundant packets are generated for every 10 data packets, even if 2 original data packets are lost, they can be recovered through the redundant packets and the remaining original packets, reducing screen freezes and audio interruptions.

[0045] Error concealment is performed on data packets lost during transmission. When data packet loss causes image damage, for image data, an interpolation algorithm of adjacent pixels is used to estimate the value of the lost pixels; for audio data, error concealment is performed by repeating or smoothly transitioning adjacent audio frames. For example, in a video, if part of the data of a frame is lost, the similarity of adjacent frames is used to fill in pixels to make the playback effect as smooth as possible and reduce obvious distortion. Selective retransmission is performed on data loss of key frames or important audio clips that are detected. When key data loss is detected and cannot be effectively restored through forward error correction and error concealment, the sender is selectively requested to retransmit the lost data packet without significantly affecting the low-latency performance. For example, the I frame of the video or the beginning of the audio is retransmitted to ensure accurate reception of important information.

[0046] The buffer size is adaptively adjusted based on whether the network condition is good or not. When the network condition is good, the buffer size is reduced to reduce latency; when the network fluctuates, the buffer is appropriately increased to cope with short-term packet delays or losses, but the buffer upper limit is strictly controlled to avoid excessive delays. For example, the initial buffer size is set to 500ms of data volume. When the packet delay is continuously detected to exceed a certain threshold, the buffer is gradually increased to 1000ms, but not more than 1500ms.

[0047] When optimizing the division and encapsulation of RTP data packets, by comparing the lengths of the timestamp, sequence number, payload format, adapted network conditions and statistical information fields, the fields of the preset length are divided to the front for optimization, and the meaningless or redundant information in each field is deleted before encapsulation. The preset length is set according to the actual data format of different formats, for example, the preset length is a shorter length of 4 characters, 2 characters, etc.

[0048] During the encoding process, the image parameters are automatically adjusted according to the playback environment and the playback status of the terminal device to enhance the image quality. The playback environment includes the ambient light and brightness, as well as the playback quality, screen resolution, and screen reflection of the terminal device. Based on this, the image parameters are automatically adjusted to enhance the contrast when the ambient light and brightness are high, making the image quality easier to see.

[0049] During the encoding and decoding process, multiple audio formats and channels are set. The specific audio formats and channels are set according to actual needs, and it has an audio equalization adjustment function.

[0050] The trial control operations are carried out with simple interface operation buttons, and the trial control operations are encrypted, automatically recovered and fault-tolerant. Through the convenient operation interface, various information in the trial can be recorded quickly and accurately, including statements of all parties, evidence presentation, trial process, etc.

[0051] The process text information obtained by real-time recording is synchronously obtained in chronological order, and the process text information is intelligently displayed to different source parties in combination with the external environment.

[0052] The cloud court information encoding and decoding of the first embodiment of the present invention performs some information interaction work in advance before the interaction work, and can be processed on the basis of the equipment originally built by various judicial departments, saving the equipment cost of the overall transformation. It can also perform information interaction after encoding or decoding data from different sources in advance, so that the information interaction is minimally disturbed by encoding or decoding, preventing delays and freezes, and preventing interruptions in the interaction process.

[0053] Embodiment 2

[0054] The streaming media data encoding and decoding method for court trial is different from the first embodiment in that it also includes the following contents:

[0055] The process text information obtained by real-time recording is synchronously acquired in chronological order, and the process text information is intelligently displayed to different information sources in combination with the external environment. The process text information includes statements, opinions, and concluding language information of multiple parties during the interaction process.

[0056] Intelligent display assesses the cultural level of the parties according to their age, work, and registered residence address, and highlights the central idea information in the interactive data based on the cultural level. The highlighting is displayed through color identification and font size. When the cultural level is lower than the preset requirements, the preset requirements are set according to the actual situation. For example, if the party is literate and has the ability to distinguish the basic legal knowledge, the central idea information is translated into simple information that is easy to understand and highlighted. For parties over 50 years old, the font size of the document displayed synchronously will be automatically increased by 2 font sizes to provide a clearer and more comfortable reading experience. For example, a document originally in size 12 will be adjusted to size 14 so that older parties can read it easily. This design fully takes into account the differences in vision among people of different age groups and reflects the care for older parties.

[0057] Since the source parties of the cloud-based shared court are uncertain and their cultural levels vary greatly, there are large differences in their ability to view and understand various interactive information. Therefore, through intelligent display, it can adapt to the use of a wider range of people, so that the information interaction of the entire cloud court can proceed smoothly and improve the efficiency of the judicial process. From font size adjustment, color contrast adjustment, voice broadcast, to key content annotation, multi-language support and emotional comfort and other functions, the special needs and psychological states of different parties are fully taken into account, making the product full of humanistic care. The simple operation process and thoughtful guidance prompts allow all kinds of people to feel respected and convenient during the trial.

[0058] Embodiment 3

[0059] The streaming media data encoding and decoding method used in court trial is different from the embodiment 1 in that Figure 2 As shown, it also includes identifying the information interaction interface identifier of each information source according to a preset priority level, determining the regional information and information conversion equipment of each information source, and dynamically adjusting the reserved time for data encoding or decoding according to the priority level.

[0060] The information interaction interface may be an interface of different devices of different information sources, such as HDMI, DVI, VGA, etc. The information interaction interface identifiers are one-to-one corresponded through letter-plus-number identifiers, and the corresponding identifier relationship information is stored in the internal memory. The regional information of the information source may be the region where the remote network is located, such as a region far away from the location of the shared court. The information conversion device may be the conversion device of the information source at the location of the regional information, which may vary greatly due to different regions or units and enterprises.

[0061] The priority level is determined according to the time taken to encode and decode the data of the information interaction interface from long to short and from difficult to easy. The priority level is preset according to the data encoding or decoding difficulty and time corresponding to the information interaction interface, and the greater the encoding difficulty or decoding time, the higher the priority.

[0062] The reserved time is dynamically adjusted according to the priority level of each information interaction interface. The higher the priority level, the longer the setting. The reserved time can be set to cover the time required for encoding or decoding of each interface. When the priority level of the information interaction interface changes, the reserved time changes adaptively.

[0063] Data encoding or decoding is performed within the reserved time, and the remaining time required for data encoding or decoding is calculated in real time. The calculation of the remaining time is performed according to the existing method and will not be repeated here. Determine whether the remaining time is less than the interaction time of the encoded or decoded data. The interaction time is the expected time for one party to reply and feedback after the information is played, as well as the shortest time required for multi-party information interaction. If so, the environmental conditions at the regional information are obtained in real time. The environmental conditions include historical environmental information and real-time environmental information. The environmental conditions can be historical or real-time weather conditions, such as heavy rain, strong wind, sunny days, light rain, etc. The abnormal probability of the information conversion equipment is judged according to the real-time environmental conditions; if not, the information interaction is performed after the reserved time ends.

[0064] The abnormality probability is determined based on the real-time environmental information and whether the historical environmental information is abnormal. For example, if the historical environmental information is moderate rain for many times and no equipment abnormality is caused, when the real-time environment is moderate rain, the abnormality probability is determined to be 40%; for another example, if the historical environmental information is heavy rain for many times and 80% of the heavy rain weather causes equipment abnormality, when the real-time environment is heavy rain, the abnormality probability is determined to be 75%; for another example, if the historical environmental information is heavy rain for many times and 10% of the heavy rain weather causes equipment abnormality, when the real-time environment is heavy rain, the abnormality probability is determined to be 20%.

[0065] When the abnormal probability is less than the preset threshold, the preset threshold is set according to actual needs. For example, the preset threshold is set to 60%, and information interaction is performed before the reserved time ends. When the abnormal probability is greater than the preset threshold, information interaction is performed after the reserved time ends.

[0066] The scheme of the first embodiment of the present invention sets different reserved times for encoding or decoding of data from different regions and information conversion devices, so that the encoding or decoding work can be carried out within the reserved time before the formal interaction, which does not take up too much interaction time and the system running memory required for the interaction; at the same time, the remaining time of the encoding or decoding is compared with the completed information interaction time. When the remaining time is less than the interaction time, that is, the remaining time required for the encoding or decoding work is less than the interaction time of the encoded or decoded content, the abnormal probability of the information conversion device being interfered with is judged according to the environmental conditions in the region, that is, the probability of the information conversion device being abnormal or damaged due to environmental influences is judged. In different situations, it is judged when to perform information interaction, and some work before the information interaction is completed to a certain extent in advance before the corresponding information interaction process is performed, thereby reducing the probability of the information interaction process being interfered with by the encoding or decoding work.

[0067] Embodiment 4

[0068] The difference between the streaming media data encoding and decoding method used in court trials and the third embodiment is that when information interaction is performed, the start and end points of the playback time of the players of the interactive data in the same area are identified, and the players, such as televisions or computers, align the start and end points of the playback time of the players in the same area to the same time point before playing. It is possible to complete part or all of the encoding or decoding work in advance, and synchronously start the playback interaction without affecting the subsequent information interaction, so as to reduce the delay in the information interaction playback process as much as possible, so that the playback time of the playback source is not delayed, and the information is played synchronously.

[0069] During playback, the emotional state of the parties concerned is identified in real time. The emotional state can be identified through facial expressions, sound decibels, and key words in speech. For example, the facial expression is identified to become angry, the sound decibels suddenly increase by 25 decibels, which is too much, and the offensive key words such as hitting and injuring, and swear words in the speech indicate emotional excitement. When the emotional state is identified as a change from normal emotion to excitement, the video is played in a gentle tone. The gentle tone can be a soft and gentle tone that makes people feel pleasant.

[0070] Using emotion recognition to trigger the playback of interactive information that can soothe people's emotions can intelligently calm emotional parties and ensure that the progress of the court review will not be disturbed or affected too much.

[0071] Embodiment 5

[0072] A method for storing streaming media data for court trials, which automatically adjusts storage location and backup strategy based on the method of embodiment 1, embodiment 2, embodiment 3, or embodiment 4 and access frequency;

[0073] Operate and feedback the video recording tasks, customize the parameters of the video recording tasks, and intelligently analyze the video content, such as analyzing the special micro-expressions or language words of the parties in the video content, automatically identify important events and abnormal situations, and provide auxiliary decision-making for the video recording tasks;

[0074] Operational management of historical video recording tasks, such as adding keyword settings for quick retrieval and query, automatic generation of video summaries of historical videos, and automatic generation of keywords and sentences in voice information using existing algorithms;

[0075] The compatible playback component automatically adjusts the video playback quality according to network conditions and device performance, and supports multi-screen playback.

[0076] Perform video synthesis, decomposition, text overlay, image overlay and special effects operations;

[0077] Set up multiple API interfaces and perform security verification on the API interfaces.

[0078] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for encoding and decoding streaming media data for court trials, characterized in that: Includes the following: Encode streaming media data in different data formats according to original data processing, H.264 encoding, NALU structure organization and encapsulation format selection, and transmit the encoded data; Decode the received encoded data and play it back; During the encoding process, the UDP-based RTP protocol performs low-latency transmission and reduces the protocol header overhead according to a preset ratio. The RTP data packets are optimized for division and encapsulation according to the characteristics of the streaming media data. The sending rate of the RTP data packets is dynamically adjusted according to the network congestion conditions and feedback from the receiving end, so as to achieve fast transmission with a real-time priority strategy.

2. The method for encoding and decoding streaming media data for court trials according to claim 1 is characterized in that: The real-time priority strategy is to process the occasionally lost data packets through a preset error correction mechanism.

3. The method for encoding and decoding streaming media data for court trials according to claim 1 is characterized in that: When optimizing the division and encapsulation of RTP data packets, by comparing the lengths of the timestamp, sequence number, payload format, adapted network conditions, and statistical information fields, the fields of preset length are divided to the front for optimization, and the meaningless or redundant information in each field is deleted before encapsulation.

4. The method for encoding and decoding streaming media data for court trials according to claim 2 is characterized in that: The preset error correction mechanism is: When sending, the original media data is encoded using the RS error correction coding algorithm; It can conceal the data errors of packets lost during transmission and selectively retransmit the data loss of key frames or important audio segments detected; The buffer size is adaptively adjusted based on whether the network condition is good or not.

5. The method for encoding and decoding streaming media data for court trials according to claim 1 is characterized in that: It also includes, during the encoding process, automatically adjusting picture parameters according to the playback environment and the playback status of the terminal device to enhance the picture quality display; During the encoding and decoding process, multiple audio formats and channels can be set, and the audio equalization adjustment function is available.

6. The method for encoding and decoding streaming media data for court trials according to claim 1, characterized in that: Also included: The trial control operation is carried out with simple interface operation buttons, and the trial control operation is encrypted, automatically recovered and fault-tolerant; The process text information obtained by real-time recording is synchronously obtained in chronological order, and the process text information is intelligently displayed to different source parties in combination with the external environment.

7. The method for encoding and decoding streaming media data for court trials according to claim 6, characterized in that: The intelligent display evaluates the cultural level of the parties according to the age, work and registered residence of the parties on the source side, determines the central idea information in the interactive data based on the cultural level and highlights it; when the cultural level is lower than the preset requirement, translates the central idea information into simple information that is easy to understand and highlights it.

8. The method for encoding and decoding streaming media data for court trials according to claim 7, characterized in that: The highlighted display is displayed through color identification, color and contrast adjustment, and font size.

9. A method for storing streaming media data for court trials, characterized in that: Includes the following: The video data of the interactive process is stored in a preset area for operation management, and the storage location and backup strategy are automatically adjusted according to the importance and access frequency of the video using a preset algorithm; Operate and feedback the video recording tasks, customize the parameters of the video recording tasks, intelligently analyze the video content, automatically identify important events and abnormal situations, and provide auxiliary decision-making for the video recording tasks; Operational management of historical video recording tasks, quick retrieval and query, and automatic generation of video summaries of historical videos; The compatible playback component automatically adjusts the video playback quality according to network conditions and device performance, and supports multi-screen playback.

10. The method for storing streaming media data for court trials according to claim 9, characterized in that: Also includes: Perform video synthesis, decomposition, text overlay, image overlay and special effects operations; Set up multiple API interfaces and perform security verification on the API interfaces.

Images