Airplane audio and video streaming media webpage synchronous playback method and device

Abstract

The invention provides an aircraft audio and video streaming media webpage synchronous playback method and device, and the method comprises the steps: setting a global timer which is responsible for monitoring the playing states of all videos; for each video, setting an independent timer for monitoring the attribute of the video; judging whether the actual playing speed is consistent with the reference speed or not by calculating the delay between the actual playing time and the reference time; setting a threshold value according to the playing speed, if the delay is greater than the threshold value, indicating that the actual playing speed is higher than the reference speed, and pausing playing and waiting; if the delay is smaller than the threshold value, the actual playing speed is slower than the reference speed, the current playing time of the video needs to be adjusted, and the index of the current frame is updated; if the current frame index is smaller than or equal to the length of the reference frame array, it shows that the video is not played completely, and under the condition, the current frame index is updated so that the next frame can be played.

Description

Technical Field

[0001] The present application belongs to the technical field of audio and video playback in ground support equipment, and specifically relates to a method and device for synchronous playback of streaming media web pages of aircraft audio and video. Background Art

[0002] Streaming media technology is a network transmission technology that processes continuous image and sound information and places it on a website server. The video server then transmits each compressed package to the user's computer sequentially or in real time, allowing users to watch and listen while downloading, without having to wait until the entire file is downloaded before watching.

[0003] Streaming media technology is widely used in video live broadcast, video on demand and other fields. It enables users to watch various types of live events in real time, enjoy high-definition, seamless video playback experience, and participate in remote communication and writing. A streaming media web player is a tool that can play streaming media content in a web environment.

[0004] The current airborne system needs to be compatible with multiple devices and video formats, while the streaming media web page playback system can convert audio and video content of various types and formats into a unified specified video format. The web page streaming media playback system can meet the playback requirements of different devices and audio and video formats. Therefore, a method for synchronous playback of audio and video data recorded on the airborne of multiple models and aircraft after flight is proposed. Summary of the invention

[0005] The purpose of this application is to propose and implement the synchronous playback of multiple videos on multiple platforms and systems and other extended functions. Implement a web streaming media player system. The streaming media platform uses QT, FFMpeg, Express, Vue and SQLite3 database as the technical basis, transcodes the original audio and video data into audio and video data that can be used by the browser, and builds a web server for web page interface and processing flow design, realizes synchronous playback of streaming media videos, storage management, permission management, and realizes command, information reception and corresponding processing functions through the network UDP protocol.

[0006] In a first aspect, the present application provides a method for synchronous playback of streaming media web pages of aircraft audio and video, the method comprising:

[0007] A global timer is set to monitor the playback status of all videos; wherein the global timer is used to periodically update the playback time and progress bar time;

[0008] For each video, a separate timer is set to monitor the attributes of the video; wherein the attributes include the current timestamp and the timestamp corresponding to the current frame;

[0009] By calculating the delay between the actual playback time and the reference time, it is determined whether the actual playback speed is consistent with the reference speed;

[0010] A threshold is set according to the playback speed. If the delay is greater than the threshold, it means that the actual playback speed is faster than the reference speed. In this case, the playback is paused and waiting. If the delay is less than the threshold, it means that the actual playback speed is slower than the reference speed. In this case, the current playback time of the video needs to be adjusted and the index of the current frame needs to be updated.

[0011] If the current frame index is less than or equal to the length of the reference frame array, it means that the video has not been played completely. In this case, update the current frame index to play the next frame. If the current frame index exceeds the length of the reference frame array, it means that the video has been played completely. Clear the listener and reset related properties to ensure that the playback status is managed correctly.

[0012] Preferably, before setting a global timer to monitor the playback status of all videos, the method further includes:

[0013] The transcoding module extracts H.264 stream from the video in a given format, and then converts the extracted H.264 stream into MP4 format in a multi-threaded manner;

[0014] The frame insertion module performs frame insertion operation on the H.264 video to maintain a stable frame rate, and then converts the H.264 video after frame insertion into an MP4 video source.

[0015] Preferably, the method further comprises:

[0016] Storing the MP4 video source in a database;

[0017] The browser playing module obtains the MP4 video source from the database and plays it.

[0018] Preferably, the method further comprises:

[0019] During the synchronous playback of audio and video, the playback jump will be controlled accordingly, and the jump mode will be set for multiple audio and video channels.

[0020] In a second aspect, the present application also provides a streaming media web page synchronous playback device for aircraft audio and video, the device comprising:

[0021] A global timer is responsible for monitoring the playback status of all videos; wherein the global timer is used to periodically update the playback time and progress bar time;

[0022] A separate timer for monitoring the attributes of each video; wherein the attributes include a current timestamp and a timestamp corresponding to a current frame;

[0023] A calculation module, used to determine whether the actual playback speed is consistent with the reference speed by calculating the delay between the actual playback time and the reference time;

[0024] The first processing module is used to set a threshold value according to the playback speed. If the delay is greater than the threshold value, it indicates that the actual playback speed is faster than the reference speed, and the playback is paused and waited; if the delay is less than the threshold value, it indicates that the actual playback speed is slower than the reference speed, and the current playback time of the video needs to be adjusted, and the index of the current frame needs to be updated;

[0025] The second processing module is used to update the current frame index if the current frame index is less than or equal to the length of the reference frame array, indicating that the video has not been played completely. In this case, the current frame index is updated to play the next frame; if the current frame index exceeds the length of the reference frame array, it means that the video has been played completely, clear the listener, reset related properties, and ensure that the playback status is correctly managed.

[0026] Preferably, the device further comprises:

[0027] The transcoding module is used to extract H.264 bitstream from a video of a given format, and then convert the extracted H.264 bitstream into MP4 format in a multi-threaded manner;

[0028] The interpolation module is used to perform interpolation operation on H.264 video to maintain a stable frame rate, and then convert the interpolated H.264 video into MP4 video source.

[0029] Preferably, the device further comprises:

[0030] A database, used to store the MP4 video source;

[0031] The browser playing module is used to obtain the MP4 video source from the database and play it.

[0032] Preferably, the device further comprises:

[0033] The jump control module is used to control the playback jump during the audio and video synchronous playback process.

[0034] This application has the following technical effects:

[0035] 1. This method can store audio and video of multiple aircraft models and multiple flights on a server for preprocessing at the same time, and then play back the audio and video stored on the server synchronously on the user's computer device through server streaming. Each video channel is played in an independent window, and the video window can adjust the contrast and rotation angle as required. At the same time, the playback progress of all audio and video of multiple aircraft can be controlled through the same playback control toolbar, so that the aircraft status of multiple flights at a certain time point and time range in the same training mission can be intuitively observed, which significantly improves the efficiency of mission playback evaluation.

[0036] 2. This method can achieve playback control effects with external software through UDP messages, significantly improving the comprehensive evaluation efficiency of the task playback process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall framework diagram of the streaming media playback system;

[0038] Figure 2 It is a schematic diagram of the interaction process of the streaming media playback system software;

[0039] Figure 3 Schematic diagram of a listener for synchronous control;

[0040] Figure 4 Schematic diagram of a single-channel monitor for synchronous control;

[0041] Figure 5 This is a flowchart of audio and video playback jump control. DETAILED DESCRIPTION

[0042] The present invention implements transcoding software to transcode the original video data into an audio and video format suitable for web page playback. Users access the web page through a browser and play the streaming media audio and video synchronously. The software design adopts a universal modular design, fully considering the requirements of universality, cross-platform, ease of use, etc., which is convenient for subsequent software expansion and upgrading. The software is divided into two core modules: transcoding and browser playback:

[0043] The transcoding module is designed and developed based on Qt. It is a comprehensive integrated software that integrates basic transcoding, batch processing, automatic transcoding, real-time communication, database access and other functions.

[0044] The browser playback module realizes the function of synchronous playback of multiple audio and video channels, designs web pages through vue+element UI, builds the back-end program through the express framework, accesses the database, connects the front-end web page with the database, and realizes the complete playback process of transcoding - database - browser.

[0045] The present invention mainly includes two core modules: a transcoding module and a browser playback module. The two core modules are connected through a database. The transcoding module stores the transcoded streaming media data in the database. The browser web page accesses the database to obtain the streaming media resource path and makes a request to the server. After receiving the request, the server will find the corresponding MP4 file according to the requested URL, and return the file content to the browser through an HTTPS response. The received media data is synchronized and the playback is controlled to realize the browser synchronous playback of streaming media audio and video. The specific processing flow is as follows: Figure 2 shown.

[0046] The following is based on the attached Figure 1 -Attached Figure 5 The present invention is described in further detail.

[0047] The process of implementing the synchronous playback solution of the present invention is as follows:

[0048] Firstly, the H.264 code stream is extracted from the video in a given format through the transcoding module, and then the extracted H.264 code stream is converted into MP4 format in a multi-threaded manner.

[0049] However, there is a problem of uncertain frame rate under a given format. To solve this problem, the module introduces an interpolation module. The main function of the interpolation module is to interpolate H.264 video to maintain a stable frame rate, and then convert the interpolated H.264 video into an MP4 video source. The design goal of this module is to provide an efficient and stable transcoding process to ensure that the output MP4 video source meets the expected requirements in terms of frame rate and format.

[0050] When the transcoding process is completed, the server will store the pre-processed aircraft audio and video information, so a method for synchronous playback of audio and video for web page playback is designed. The implementation method is as follows:

[0051] 1) Global playback status monitoring timer: A global timer is set in the system to monitor the playback status of all videos. The timer periodically updates the playback time and progress bar time so that users can understand the current playback progress in real time.

[0052] 2) Individual video playback attribute monitoring timer: For each video, a separate timer is set to monitor the relevant attributes of the video. These attributes include the current timestamp (currentPTS) and the timestamp corresponding to the current frame (currentFramePTS).

[0053] 3) Delay calculation and adjustment: By calculating the delay between the actual playback time and the reference time, it can be determined whether the actual playback speed is consistent with the reference speed. The delay threshold is set according to the playback speed. When it is less than 4 times the speed, the threshold is ±200ms, and when it is 8 times to 32 times the speed, the threshold is ±40ms. If the delay is greater than the threshold, it means that the actual playback speed is faster than the reference speed. In this case, pause the playback and wait. If the delay is less than the threshold, it means that the actual playback speed is slower than the reference speed. It is necessary to adjust the current playback time of the video and update the index of the current frame.

[0054] 4) If the current frame index is less than or equal to the length of the reference frame array, the video has not yet been played. In this case, update the current frame index so that the next frame can be played. If the current frame index exceeds the length of the reference frame array, the video has been played completely, so you need to clear the listener and reset the relevant properties to ensure that the playback state is managed correctly.

[0055] (3) Multi-channel video jump control

[0056] During the audio and video synchronous playback process, the playback jump control will be carried out accordingly. For the multi-channel audio and video jump mode, the flow chart is as follows Figure 5 shown.

[0057] 1) Get the video element object.

[0058] 2) Determine whether there is an event display at the time of the jump. If there is an event display, pause the playback after the jump.

[0059] 3) Calculate the difference (diff) between the video start time and the progress bar start time to determine whether delayed playback is required.

[0060] 4) If the current video playback time is greater than 0, it means that the video has started playing. Perform corresponding operations according to the jump time:

[0061] a. If diff is equal to 0, it means that the video is played without delay, and the current playback time of the video is directly set to the given time (time).

[0062] b. If diff is not 0, it means the video is delayed. Continue to determine whether the video starts playing at the jump point.

[0063] c. If the video has started playing and the jump time is before the video starts playing, perform the following operations: ① Reset the end playback status of the video; ② Set the current playback time of the video to 0, that is, restart the playback; ③ Clear the frame monitoring timer of the video; ④ Clear the previously set delayed playback timer; ⑤ Reset the delayed playback timer according to the delay time and playback speed, and perform the playback operation after the delay ends.

[0064] d. If the jump time is after the video is played, do the following: ① Cancel the previously set delayed playback timer; ② Set the current playback time of the video to the jump time minus the delay time, and fine-tune the time difference of the delayed playback; ③ Update the current frame of the video.

[0065] (3) Playback control based on UDP communication

[0066] During the mission training, relevant software will be used to analyze and evaluate the flight process. Therefore, linking the player with external software can greatly improve the efficiency of flight mission evaluation. For this purpose, a UDP communication control function is designed. The server receives external UDP messages through UDP Socket and sends them to the client through WebSocket. The client sends messages to the server through WebSocket, and the server sends the message to the external UDP Socket. In this way, the process of UDP message transmission through WebSocket is realized. The client and server realize two-way communication through WebSocket, and realize UDP communication with the outside through UDP Socket.

Claims

1. A method for synchronous playback of streaming media web pages of aircraft audio and video, characterized in that: The method comprises: A global timer is set to monitor the playback status of all videos; wherein the global timer is used to periodically update the playback time and progress bar time; For each video, a separate timer is set to monitor the attributes of the video; wherein the attributes include the current timestamp and the timestamp corresponding to the current frame; By calculating the delay between the actual playback time and the reference time, it is determined whether the actual playback speed is consistent with the reference speed; A threshold is set according to the playback speed. If the delay is greater than the threshold, it means that the actual playback speed is faster than the reference speed. In this case, the playback is paused and waiting. If the delay is less than the threshold, it means that the actual playback speed is slower than the reference speed. In this case, the current playback time of the video needs to be adjusted and the index of the current frame needs to be updated. If the current frame index is less than or equal to the length of the reference frame array, it means that the video has not been played completely. In this case, update the current frame index to play the next frame. If the current frame index exceeds the length of the reference frame array, it means that the video has been played completely. Clear the listener and reset related properties to ensure that the playback status is managed correctly.

2. The method according to claim 1, characterized in that Before setting a global timer to monitor the playback status of all videos, the following steps are also included: The transcoding module extracts H.264 bitstream from the video in a given format, and then converts the extracted H.264 bitstream into MP4 format in a multi-threaded manner; The frame insertion module performs frame insertion operation on the H.264 video to maintain a stable frame rate, and then converts the interpolated H.264 video into an MP4 video source.

3. The method according to claim 2, characterized in that The method further comprises: Storing the MP4 video source in a database; The browser playing module obtains the MP4 video source from the database and plays it.

4. The method according to claim 3, characterized in that The method further comprises: During the synchronous playback of audio and video, the playback jump will be controlled accordingly, and the jump mode will be set for multiple audio and video channels.

5. A streaming media web page synchronous playback device for aircraft audio and video, characterized in that: The device comprises: A global timer is responsible for monitoring the playback status of all videos; wherein the global timer is used to periodically update the playback time and progress bar time; A separate timer for monitoring the attributes of each video; wherein the attributes include a current timestamp and a timestamp corresponding to a current frame; A calculation module, used to determine whether the actual playback speed is consistent with the reference speed by calculating the delay between the actual playback time and the reference time; The first processing module is used to set a threshold value according to the playback speed. If the delay is greater than the threshold value, it indicates that the actual playback speed is faster than the reference speed, and the playback is paused and waited; if the delay is less than the threshold value, it indicates that the actual playback speed is slower than the reference speed, and the current playback time of the video needs to be adjusted, and the index of the current frame needs to be updated; The second processing module is used to update the current frame index if the current frame index is less than or equal to the length of the reference frame array, indicating that the video has not been played completely. In this case, the current frame index is updated to play the next frame; if the current frame index exceeds the length of the reference frame array, it means that the video has been played completely, clear the listener, reset related properties, and ensure that the playback status is correctly managed.

6. The device according to claim 5, characterized in that The device also includes: The transcoding module is used to extract H.264 bitstream from a video of a given format, and then convert the extracted H.264 bitstream into MP4 format in a multi-threaded manner; The interpolation module is used to perform interpolation operation on H.264 video to maintain a stable frame rate, and then convert the interpolated H.264 video into MP4 video source.

7. The device according to claim 6, characterized in that The device also includes: A database, used to store the MP4 video source; The browser playing module is used to obtain the MP4 video source from the database and play it.

8. The device according to claim 7, characterized in that The device also includes: The jump control module is used to control the playback jump during the audio and video synchronous playback process.

Images