Display device, video data transmission method and storage medium
By adjusting the buffer queue preset value and dynamically creating the buffer queue, the playback lag caused by the full video buffer queue and the idle audio buffer queue is solved, and the smooth playback and performance improvement of audio and video data is achieved.
Patent Information
- Application Number
- CN202210014333.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-04
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-01-04
AI Technical Summary
In the display device, when the video buffer queue is full and the audio buffer queue is idle, video data cannot be stored, resulting in stuttering playback, especially when the high-resolution video data exceeds the maximum data volume of the buffer queue in a short period of time.
By adjusting the preset buffer amount of the video buffer queue and the target buffer queue, dynamically create a buffer queue to store video data, and store the dynamic buffer queue data into the video buffer queue within the audio and video synchronization time difference to avoid playback stuttering.
It effectively avoids playback lag, ensures smooth playback of audio and video data, and improves the performance of the player without increasing the system storage resource usage.
Smart Images

Figure CN114245180B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of display technology, and in particular to a display device, a video data transmission method and a storage medium. Background Art
[0002] In the player of the display device, separate buffer queues are set for audio data and video data. Each buffer queue has a preset buffer amount for buffered data. The preset buffer amount is used to indicate the maximum amount of data that can be buffered by the current buffer queue. When the buffered data in the video buffer queue reaches the maximum amount of data set for the video buffer queue, new video data cannot be stored in the video buffer queue. It can only wait until part of the data in the current video buffer queue is consumed before it can continue. For some special sources, the distribution of audio data and video data is uneven. There may be a situation where a certain section of data is all video data. And if it is high-resolution video data, the video data in a very short time may exceed the maximum amount of data allowed by the video buffer queue. At this time, the video buffer queue will be full of video data, but there will be idle audio buffer queues.
[0003] However, in the above case, before the video data is sent to the video buffer queue, the subsequent data cannot be read. Therefore, when the video buffer queue is full but no audio data is read, the audio and video data cannot be played, resulting in playback jams. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a display device, a video data transmission method and a storage medium, which can allocate the remaining idle buffer controls in other buffer queues to the video buffer queue, so that the video buffer queue can continue to buffer the video data. After reading the audio data, the audio and video data can be played normally to avoid playback jams.
[0005] In order to achieve the above purpose, the technical solution provided by the embodiment of the present invention is as follows:
[0006] In a first aspect, a display device is provided, comprising:
[0007] The controller is configured to: obtain first video data;
[0008] When the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue and the buffered data in the target buffer queue has not reached a second preset buffer amount of the target buffer queue, setting the first preset buffer amount of the video buffer queue to a third preset buffer amount and setting the second preset buffer amount of the target buffer queue to a fourth preset buffer amount;
[0009] Storing the first video data in the video buffer queue;
[0010] Among them, the third preset buffer amount is increased by a preset amount compared to the first preset buffer amount, the fourth preset buffer amount is reduced by the preset amount compared to the second preset buffer amount, and the target buffer queue is any queue in the other buffer queues, and the other buffer queues are used to buffer data other than video data.
[0011] As an optional implementation of the embodiment of the present invention, the controller is further configured as follows:
[0012] After acquiring the first video data, when the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue, and the buffered data in other buffer queues have reached preset buffer amounts of their respective buffer queues, creating a dynamic buffer queue;
[0013] The first video data is stored in the dynamic buffer queue.
[0014] As an optional implementation of the embodiment of the present invention, the target buffer queue is an audio buffer queue, and the controller is further configured to:
[0015] After acquiring the first video data, when the buffered data in the video buffer queue has reached the first preset buffer amount of the video buffer queue and the buffered data in the audio buffer queue has reached the second preset buffer amount, creating a dynamic buffer queue;
[0016] The first video data is stored in the dynamic buffer queue.
[0017] As an optional implementation of the embodiment of the present invention, the controller is further configured as follows:
[0018] After creating the dynamic buffer queue, obtaining second video data;
[0019] The second video data is directly stored in the dynamic buffer queue.
[0020] As an optional implementation of the embodiment of the present invention, the controller is further configured as follows:
[0021] After the dynamic buffer queue is created, the synchronization time difference of the audio and video is calculated in real time; wherein the synchronization time difference is the time difference between the display timestamp of the latest video data and the display timestamp of the latest audio data in the video buffer queue;
[0022] When the synchronization time difference is less than or equal to a preset duration, storing the data in the dynamic buffer queue into the video buffer queue;
[0023] The preset duration is smaller than the maximum synchronization time difference of the video buffer queue, and the maximum synchronization time difference is the time difference between the timestamp of the earliest video data and the timestamp of the latest video data in the video buffer queue.
[0024] As an optional implementation of the embodiment of the present invention, the controller is further configured to: cancel the dynamic buffer queue when all the data in the dynamic buffer queue has been stored in the video buffer queue.
[0025] As an optional implementation of the embodiment of the present invention, the controller is further configured as follows:
[0026] After creating the dynamic buffer queue, monitoring the amount of buffered data in the video buffer queue in real time;
[0027] When the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount, storing the data in the dynamic buffer queue into the video buffer queue;
[0028] Wherein, the target buffer amount is smaller than the first preset buffer amount.
[0029] In a second aspect, a video data transmission method is provided, comprising:
[0030] Acquire first video data;
[0031] When the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue and the buffered data in the target buffer queue has not reached a second preset buffer amount of the target buffer queue, setting the first preset buffer amount of the video buffer queue to a third preset buffer amount and setting the second preset buffer amount of the target buffer queue to a fourth preset buffer amount;
[0032] Storing the first video data in the video buffer queue;
[0033] Among them, the third preset buffer amount is increased by a preset amount compared to the first preset buffer amount, the fourth preset buffer amount is reduced by the preset amount compared to the second preset buffer amount, and the target buffer queue is any queue in the other buffer queues, and the other buffer queues are used to store data other than video data.
[0034] As an optional implementation of the embodiment of the present invention, the method further includes:
[0035] After acquiring the first video data, when the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue, and the buffered data in the other buffer queues have reached preset buffer amounts of their respective buffer queues, creating a dynamic buffer queue;
[0036] The first video data is stored in the dynamic buffer queue.
[0037] As an optional implementation of the embodiment of the present invention, the method further includes:
[0038] After acquiring the first video data, when the buffer data in the video buffer queue has reached the first preset buffer amount of the video buffer queue and the buffer data in the audio buffer queue has reached the second preset buffer amount, creating a dynamic buffer queue;
[0039] The first video data is stored in the dynamic buffer queue.
[0040] As an optional implementation of the embodiment of the present invention, the method further includes:
[0041] After creating the dynamic buffer queue, obtaining second video data;
[0042] The second video data is directly stored in the dynamic buffer queue.
[0043] As an optional implementation of the embodiment of the present invention, the method further includes:
[0044] After the dynamic buffer queue is created, the synchronization time difference of the audio and video is calculated in real time; wherein the synchronization time difference is the time difference between the display timestamp of the latest video data and the display timestamp of the latest audio data in the video buffer queue;
[0045] When the synchronization time difference is less than or equal to a preset duration, storing the data in the dynamic buffer queue into the video buffer queue;
[0046] The preset duration is smaller than the maximum synchronization time difference of the video buffer queue, and the maximum synchronization time difference is the time difference between the timestamp of the earliest video data and the timestamp of the latest video data in the video buffer queue.
[0047] As an optional implementation of the embodiment of the present invention, the method further includes:
[0048] When all the data in the dynamic buffer queue has been stored in the video buffer queue, the dynamic buffer queue is cancelled.
[0049] As an optional implementation of the embodiment of the present invention, the method further includes:
[0050] After creating the dynamic buffer queue, monitoring the amount of buffered data in the video buffer queue in real time;
[0051] When the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount, storing the data in the dynamic buffer queue into the video buffer queue;
[0052] Wherein, the target buffer amount is smaller than the first preset buffer amount.
[0053] In a third aspect, the present invention provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the video data transmission method shown in the second aspect is implemented.
[0054] In a fourth aspect, the present invention provides a computer program product, characterized in that it includes: when the computer program product runs on a computer, it enables the computer to implement the video data transmission method shown in the second aspect.
[0055] The display device and video data transmission method provided by the embodiments of the present invention can, when the buffered data in the video buffer queue has reached the currently allowed first preset buffer amount, if it can be determined in other buffer queues that the buffered data has not reached the target buffer queue of the second preset buffer amount, thus confirming that although the video buffer queue is full, the target buffer queue is not full. At this time, the preset values of the buffer amounts allowed by the video buffer queue and the target buffer queue can be modified, and the preset buffer quota in the target buffer queue can be allocated to the video buffer queue, so that the video buffer queue can continue to buffer the video data, and after reading the audio data, the audio and video data can be played normally to avoid playback jams. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0058] Figure 1 A schematic diagram of the architecture of video data transmission in a player provided by an embodiment of the present invention;
[0059] Figure 2 Schematic diagrams of scenarios in some embodiments provided for embodiments of the present invention;
[0060] Figure 3 A configuration block diagram of a control device 100 provided in an embodiment of the present invention;
[0061] Figure 4 A hardware configuration block diagram of a display device 200 provided in an embodiment of the present invention;
[0062] Figure 5 A schematic diagram of software configuration in a display device 200 of one or more embodiments provided by an embodiment of the present invention;
[0063] Figure 6 The embodiment of the present invention provides Figure 1 A schematic diagram of a scenario in which the player architecture shown implements a method for transmitting video data;
[0064] Figure 7 The embodiment of the present invention provides Figure 1 The player architecture shown is another schematic diagram of a method for implementing video data transmission;
[0065] Figure 8 A method flow of a video data transmission method provided by an embodiment of the present invention Figure 1 ;
[0066] Fig. 9 A method based on the embodiment of the present invention is provided Figure 6 A schematic diagram of a video data transmission method for the illustrated scenario;
[0067] Fig.10 A method based on the embodiment of the present invention is provided Figure 7 A schematic diagram of a video data transmission method for the illustrated scenario;
[0068] Fig.11 A schematic diagram of a video data transmission method in another scenario provided by an embodiment of the present invention;
[0069] Fig.12 A method flow of a video data transmission method provided by an embodiment of the present invention Figure 2 ;
[0070] Fig.13 A method flow of a video data transmission method provided by an embodiment of the present invention Figure 3 . DETAILED DESCRIPTION
[0071] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0072] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.
[0073] In the player of the current display device, independent buffer queues are set for different data. For example, independent buffer queues are set for audio data and video data respectively. Each buffer queue has a preset buffer amount for buffered data. The preset buffer amount is used to indicate the maximum amount of data that can be buffered by the current buffer queue.
[0074] For example, Figure 1 As shown, Figure 1 A schematic diagram of a video data transmission architecture provided by an embodiment of the present invention, Figure 1 In the player, a modem 11 (DEMUX, also called a demultiplexer or demultiplexer), a buffer queue 12, an audio decoder 13, a video decoder 14, a subtitle selector 15, and a rendering module 16 are provided. It can be seen that in the buffer queue 12 in the player, an audio buffer queue corresponding to audio data, a video buffer queue corresponding to video data, and a subtitle buffer queue corresponding to a plurality of subtitle data are provided. Each subtitle buffer queue is used to buffer different types of subtitle data. For example, one subtitle buffer queue can be used to buffer English subtitle data represented by subtitle 1, one subtitle buffer queue can be used to buffer Chinese subtitle data represented by subtitle 2, and another subtitle buffer queue can be used to buffer Korean subtitle data represented by subtitle 2. After the modem 11 obtains the multimedia file, it can demultiplex the audio data, video data and subtitle data, and store these data in corresponding data queues for buffering. Thereafter, the audio data and video data are decoded by the audio decoder 13 and the video decoder 14 respectively, and the subtitle selector 15 selects the corresponding subtitle data. Then, the decoded audio data and video data, as well as the corresponding subtitle data selected by the subtitle selector 15 are rendered by the rendering module 606, so that a multimedia screen with video screen, sound and subtitles can be displayed.
[0075] When setting the buffer queue, a preset buffer volume can be pre-set according to the actual situation of system resources. After the data volume of a buffer queue reaches the preset buffer volume, new data cannot be stored any more, and the data in the current buffer queue can only be stored after part of it is consumed. Generally, video data accounts for the largest proportion in multimedia files, and the data volume of audio data and subtitle data accounts for a relatively small proportion. In addition, video data and audio data are normally stored in an interlaced manner, and data with similar presentation time stamps (PTS) are also stored adjacently in physical locations to ensure that audio and video data can be synchronized and played smoothly.
[0076] For some special multimedia data, the distribution of audio and video data is not uniform enough. It may happen that the data received in a certain period of time is all video data. If it is a high-resolution video (such as a 4K source), the video data in a very short time may exceed the maximum value allowed by the video buffer queue (i.e., the preset buffer amount). At this time, the video buffer queue is full and no new video data can be stored, while only a small amount of data in the audio buffer queue will be completely empty. Before the data read in the modem 11 is analyzed and demultiplexed, it cannot be determined whether it is audio data or video data. Therefore, the two operations of reading data from the data stream and sending data to the buffer queue are performed in the same thread. Before the operation of sending data to the buffer queue is completed, the next data cannot be read. Therefore, in the case of uneven distribution of video data and audio data, if the video data buffered in the video buffer queue reaches the maximum value, the data sending action of the modem will stop executing and wait until the video buffer queue has storage space that can store video data, and then return to read the next data. In fact, because the modem cannot send video data into the video buffer queue, it cannot continue to demultiplex subsequent data. In this way, there is no audio data to send, which will cause the audio data and video data to be out of sync, and the video data cannot be consumed. The whole process enters an infinite loop and playback cannot proceed. As the video resolution increases, the probability of such problems will continue to increase.
[0077] In some embodiments, this problem can be solved by increasing the maximum buffer threshold of the video buffer queue. However, if only the maximum buffer threshold of the video buffer queue is increased, the requirement for hardware resources will be directly increased.
[0078] In order to solve the above problems, the embodiments of the present invention further provide some embodiments, in which:
[0079] First, when the video buffer queue is full and the remaining buffer queues are empty, the buffer space in other unused buffer queues can be temporarily borrowed to expand the buffer space of the video buffer queue to receive more video data, prevent the modem's video data retrieval and sending threads from being blocked, and allow playback to proceed normally without occupying additional system storage space.
[0080] Furthermore, if the allocated buffer queues are full, dynamic buffer queues can be enabled in the modem, and new video data is no longer sent to the system buffer queue, but is temporarily buffered locally in the modem and placed in a dynamic buffer queue. At the same time, the video data and audio data are synchronized. After the video data and audio data are synchronized to a certain time difference range, the audio and video data are sent normally. This processing method adopts a dynamic buffer strategy, and only applies for additional memory for the next level of caching when it is really needed. Under normal circumstances, it does not need to occupy system memory resources, which ensures that special sources (i.e., sources with uneven distribution of audio data and video data) will not be stuck within the range allowed by hardware resources, and will not occupy additional system resources under normal circumstances.
[0081] Figure 2 Schematic diagram of scenes in some embodiments provided by the present invention. Figure 2 As shown, the user can operate the display device 200 through the smart device 300 or the control apparatus 100 to play audio and video resources on the display device 200. The audio and video resources include various types of data, such as video data, audio data, and various subtitle data.
[0082] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user may input user commands by pressing buttons on the remote controller, voice input, control panel input, etc. to control the display device 200. In some embodiments, a mobile terminal, a tablet computer, a computer, a laptop computer, and other smart devices may also be used to control the display device 200.
[0083] In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and realize connection communication through the network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 also communicates data with the server 400 through various communication methods. The display device 200 can be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 can provide various content and interactions to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing a broadcast receiving television function, the display device 200 can also provide an intelligent network TV function that provides a computer support function.
[0084] Figure 3 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 3 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play the role of an interactive intermediary between the user and the display device 200. The communication interface 130 is used to communicate with the outside, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button or an alternative module.
[0085] Figure 4 FIG. 2 shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment. Figure 4 The display device 200 shown includes: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, etc. Among them, the controller 250 includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.
[0086] In some embodiments, the above-mentioned display device is a terminal device with a display function, such as a television, a mobile phone, a computer, a learning machine, etc.
[0087] In some embodiments, the controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. The user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.
[0088] An output interface (display 260 and / or audio output interface 270), configured to output user interaction information;
[0089] The communicator 220 is used to communicate with the server 400 or other devices.
[0090] In some embodiments, the controller 250 can use other buffer queues that have been applied for. When the video buffer queue is full and the remaining other buffer queues are free, the controller 250 can temporarily borrow space in other buffer queues that are not used to expand the space of the video buffer queue to receive more data, prevent the data fetching and sending thread of the modem from being blocked, and allow the playback to proceed normally without occupying additional system storage space. The implementation method may include: the controller 250 obtains the first video data; when the buffer data in the video buffer queue has reached the first preset buffer amount of the video buffer queue, and the buffer data in the target buffer queue has not reached the second preset buffer amount of the target buffer queue, the first preset buffer amount of the video buffer queue is set to the third preset buffer amount, and the second preset buffer amount of the target buffer queue is set to the fourth preset buffer amount; the first video data is stored in the video buffer queue; wherein the third preset buffer amount is increased by a preset amount compared to the first preset buffer amount, and the fourth preset buffer amount is reduced by the preset amount compared to the second preset buffer amount, the target buffer queue is any queue in the other buffer queues, and the other buffer queues are used to buffer other data other than the video data.
[0091] In some embodiments, the controller 250 may also create a dynamic buffer queue when no other buffer queues have free buffer space. The implementation may include: after acquiring the first video data, when the buffer data in the video buffer queue has reached the first preset buffer amount of the video buffer queue, and the buffer data in the other buffer queues have reached the preset buffer amounts of their respective buffer queues, creating a dynamic buffer queue; and storing the first video data in the dynamic buffer queue.
[0092] In some embodiments, the controller 250 may also create a dynamic buffer queue when there is no free space in the audio buffer queue. The implementation method may include: after acquiring the first video data, the controller 250 creates a dynamic buffer queue when the buffer data in the video buffer queue has reached the first preset buffer amount of the video buffer queue and the buffer data in the audio buffer queue has reached the second preset buffer amount; and stores the first video data in the dynamic buffer queue.
[0093] In some embodiments, after the controller 250 creates the dynamic buffer queue, subsequent data can be directly stored in the dynamic buffer queue. That is, the controller 250 is also used to obtain the second video data after creating the dynamic buffer queue; and store the second video data directly in the dynamic buffer queue.
[0094] In some embodiments, the controller 205 can also monitor the synchronization time difference between the video data and the audio data in real time after the dynamic buffer queue is created. When the synchronization time difference is reduced to a certain extent, it indicates that the data in the video buffer queue has been consumed, and the data in the dynamic buffer queue is stored in the video buffer queue. In another implementation, the amount of buffered data in the video buffer queue can also be monitored in real time after the dynamic buffer queue is created. After the amount of buffered data is reduced, the data in the dynamic buffer queue is stored in the video buffer queue.
[0095] One of the above-mentioned implementation methods may include: after creating a dynamic buffer queue, the controller 250 calculates the synchronization time difference of audio and video in real time; wherein the synchronization time difference is the time difference between the display timestamp of the latest video data in the video buffer queue and the display timestamp of the latest audio data; when the synchronization time difference is less than or equal to the preset duration, the data in the dynamic buffer queue is stored in the video buffer queue; wherein the preset duration is less than the maximum synchronization time difference of the video buffer queue, and the maximum synchronization time difference is the time difference between the timestamp of the earliest video data in the video buffer queue and the timestamp of the latest video data.
[0096] Another implementation of the above includes: after creating a dynamic buffer queue, monitoring the amount of buffered data in the video buffer queue in real time, and storing the data in the dynamic buffer queue into the video buffer queue when the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount; wherein the target buffer amount is less than the first preset buffer amount. Further, after the dynamic buffer queue has been fully stored in the video buffer queue, the dynamic buffer queue can be cancelled to release system resources. The implementation includes: the controller 250 can also cancel the dynamic buffer queue when the data in the dynamic buffer queue has been fully stored in the video buffer queue.
[0097] like Figure 5 As shown, Figure 5 FIG. 1 is a schematic diagram of software configuration in a display device 200 according to one or more embodiments of the present invention. Figure 5 As shown, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) (referred to as "application layer"), the application framework layer (Application Framework) (referred to as "framework layer"), the Android runtime (Androidruntime) and the system library layer (referred to as "system runtime layer"), and the kernel layer. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc. The video data transmission method provided in the embodiment of the present invention can be implemented based on the above-mentioned display device, and can be specifically implemented by a player in the display device.
[0098] like Figure 6 As shown, based on Figure 1 The schematic diagram of a scenario in which the player architecture implements the video data transmission method, in which the buffered data in the video buffer queue is full, and there is free buffer space in the remaining other buffer queues. Among them, the audio buffer queue has no data buffered temporarily. In this scenario, the video data obtained by the modem cannot be put into the video queue, and the modem cannot continue to obtain subsequent data. In this way, no audio data can be obtained to consume the video data in the video buffer queue, which will cause playback jams.
[0099] like Figure 7 As shown, based on Figure 1The player architecture shown is a schematic diagram of another scenario in which the video data transmission method is implemented. In this scenario, the buffered data in the video buffer queue is full, and there is free buffer space in the remaining other buffer queues. Among them, there is buffered data in the audio buffer queue, but the buffered data is not full. In this scenario, although the modem can continue to consume the video data in the video buffer queue, there is still a problem of uneven audio and video data subsequently. In this scenario, the video data transmission method in the embodiment of the present invention can also be used to avoid the problem of playback jams.
[0100] It should be noted that the above Figure 6 and Figure 7 The scenario shown is only an example of an audio buffer queue in other buffer queues not fully buffering data or having no buffer data. In practice, there may also be a situation where there is free buffer space in the subtitle buffer queue, which is not limited in the embodiment of the present invention.
[0101] Regarding the above Figure 6 or Figure 7 In the scenario shown, in an embodiment of the present invention, a video data transmission method can be implemented. When the buffered data in the video buffer queue is full and there is remaining buffer space in other remaining buffer queues, the buffer space in other unused buffer queues can be temporarily borrowed to expand the buffer space of the video buffer queue to receive more data, thereby preventing the data transmission thread of the modem from being blocked, and allowing the playback to proceed normally without occupying additional system storage space.
[0102] In order to explain this scheme in more detail, the following will be combined in an exemplary manner Figure 8 To explain, it is understandable that Figure 8 The steps involved in the actual implementation may include more steps, or fewer steps, and the order of these steps may also be different, so as to implement the video data transmission method provided in the embodiment of the present invention.
[0103] Figure 8 The method for transmitting video data according to the embodiment of the present invention is provided as a flowchart. Figure 8 As shown, the video data transmission method provided by the embodiment of the present invention includes the following steps S801 to S807:
[0104] S801. Obtain first video data.
[0105] After receiving the multimedia transmission packet, the player of the display device will demultiplex the received multimedia transmission packet through a modem to obtain video data and audio data, and sometimes subtitle data.
[0106] The first video data may be video data obtained after a modem demultiplexes a multimedia transmission packet.
[0107] S802: Determine whether the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue.
[0108] The first preset buffering amount may be a maximum allowable buffering data amount pre-set for the video buffer queue.
[0109] In the actual implementation process, memory space can be allocated in advance for various buffer queues in the player according to the system memory resources to configure the maximum buffer data volume allowed by various buffer queues. Figure 1 In the player architecture shown, there is a video buffer queue, an audio buffer queue and three subtitle buffer queues involved. Memory space can be allocated to the video buffer queue, the audio buffer queue and the three subtitle buffer queues respectively to configure the maximum buffer data amount allowed for the video buffer queue, the audio buffer queue and the three subtitle buffer queues.
[0110] It is understandable that, in the embodiment of the present invention, the maximum allowable buffer data amounts pre-set for different video buffer queues may be the same or different. For example: In general, the amount of video data is larger than the amount of audio data, and the amount of audio data is larger than the amount of subtitle data. Therefore, when allocating memory space, the capacity of the video buffer queue's memory space may be configured to be larger than the capacity of the audio buffer queue's memory space, and the capacity of the audio buffer queue's memory space may be larger than the capacity of the subtitle buffer queue's memory space. Exemplarily, assuming that 500M of memory space can be configured for all buffer queues of the player based on system memory resources, then in the following example: Figure 1 In the scenario shown, 300M of memory space can be allocated to the video buffer queue, 170M of memory space can be allocated to the audio buffer queue, and 10M of memory space can be allocated to each subtitle buffer queue.
[0111] In an embodiment of the present invention, when the buffered data in the video buffer queue has reached the first preset buffer amount of the video buffer queue, it means that the current video buffer queue is full, and the following S803 is executed to determine whether there are other buffer queues with free storage space, and the buffer space in the other buffer queues can be temporarily used; when the buffered data in the video buffer queue has not reached the first preset buffer amount of the video buffer queue, it means that the current video buffer queue is not full, and the following S805 can be executed to buffer the first video data.
[0112] The above-mentioned other buffer queues refer to queues set in the player for buffering other data besides video data. For example, the other buffer queues may include audio data queues, subtitle data queues, etc.
[0113] S803: Determine whether the buffered data in the target buffer queue reaches a second preset buffer amount of the target buffer queue.
[0114] The second preset buffering amount may be a maximum allowable buffering data amount pre-set for the target buffer queue. The target buffer queue may be any buffer queue among the remaining buffer queues.
[0115] For example: Figure 1 In the player architecture shown, the target buffer queue can be the audio buffer queue and any one of the three subtitle buffer queues.
[0116] In an embodiment of the present invention, when the buffered data in the target buffer queue has reached the second preset buffer amount of the video buffer queue, it means that the current target buffer queue is full, and the following S806 and S807 are executed to buffer the first video data; when the buffered data in the target buffer queue has not reached the second preset buffer amount of the target buffer queue, it means that the current target buffer queue is not full, and the following S804 and S805 can be executed, and the buffer space in the target buffer queue can be temporarily used to buffer the first video data.
[0117] S804: Set the first preset buffer size of the video buffer queue as the third preset buffer size, and set the second preset buffer size of the target buffer queue as the fourth preset buffer size.
[0118] The third preset buffering amount is larger than the first preset buffering amount by a preset amount, and the fourth preset buffering amount is smaller than the second preset buffering amount by a preset amount. In other words, the preset amount of buffer space originally belonging to the target buffer queue is given to the video buffer queue.
[0119] S805: Store the first video data into a first buffer queue.
[0120] After the setting in S804, the maximum buffer data amount allowed by the video buffer queue increases, so that the video buffer queue can store more data, and thus the first video data can be stored in the first buffer queue.
[0121] For example, in the above Figure 6In the implementation scenario shown, since the buffer space in the audio buffer queue is not occupied, that is, the audio buffer queue is empty, the preset buffer thresholds corresponding to the audio buffer queue and the video buffer queue can be adjusted, so that the maximum buffer data amount allowed by the video buffer queue increases and the maximum buffer data amount allowed by the audio buffer queue decreases, so that the video buffer queue can store more data. For example, Fig. 9 As shown, based on Figure 6 A schematic diagram of a video data transmission method for the scenario shown, Fig. 9 As shown, when the video buffer queue is full of video data and there is no audio data buffer in the audio buffer queue, after the first video data is acquired, the video data cannot be stored in the video buffer queue because the video buffer queue is full of video data. Fig. 9 In this case, part of the free buffer space in the audio buffer queue is given to the video buffer queue, so that the acquired first video data can be stored in the video buffer queue. Fig. 9 As can be seen from (b), after adjusting the preset buffer thresholds corresponding to the audio buffer queue and the video buffer queue, the buffer space of the audio buffer queue is reduced. Correspondingly, the reduced buffer space in the audio buffer queue is given to the video buffer queue, and the buffer space of the video buffer queue is increased, so that the video buffer queue can allow new video data to be stored.
[0122] For example, Figure 7 In the scenario shown, although the audio buffer queue is not occupied, there is still some free buffer space left. The preset buffer thresholds corresponding to the audio buffer queue and the video buffer queue can still be adjusted, so that the maximum buffer data amount allowed by the video buffer queue increases and the maximum buffer data amount allowed by the audio buffer queue decreases, so that the video buffer queue can store more data. For example, Fig.10 As shown, based on Figure 7 A schematic diagram of a video data transmission method for the scenario shown in FIG. Fig.10 It can be seen from (a) in that the video buffer queue can no longer store the first video data. At this time, part of the free buffer space of the audio buffer queue can be given to the video buffer queue. After that, the buffer space of the video buffer queue is increased, so that the video buffer queue can allow new video data to be stored. Fig.10 As shown in (b) in .
[0123] S806: Create a dynamic buffer queue.
[0124] S807: Store the first video data into a dynamic buffer queue.
[0125] In some embodiments, the modem in the player may locally create a dynamic buffer queue to store the first video data.
[0126] In some embodiments, after acquiring the first video data, when the buffered data in the video buffer queue has reached a first preset buffer capacity of the video buffer queue, and the buffered data in other buffer queues have reached the preset buffer capacities of their respective buffer queues, a dynamic buffer queue is created and the first video data is stored in the dynamic buffer queue.
[0127] The buffered data in the other buffer queues mentioned above have reached the preset buffering amount of each buffer queue, which can be understood as that the existing buffer queues in the player have reached the maximum buffering data amount allowed by each. Figure 1 In the player shown, if Figure 1 The data buffer queue, video buffer queue and three subtitle buffer queues have reached their respective maximum allowed buffer data amounts. At this time, there are no buffer queues that can be stored. The modem can create a dynamic buffer queue and store the first video data into the dynamic buffer queue, so that demultiplexing can continue to obtain new data.
[0128] Since there are video buffer queues and audio buffer queues in general multimedia files, and the amount of video data is usually larger than that of audio data, the free buffer space in the audio buffer queue can be used to supplement the buffer space of the video buffer queue. In some embodiments, after acquiring the first video data, when the buffer data in the video buffer queue has reached the first preset buffer amount of the video buffer queue, and the buffer data in the audio buffer queue has reached the second preset buffer amount, a dynamic buffer queue is created; and the first video data is stored in the dynamic buffer queue.
[0129] In the above implementation, the audio buffer queue, or the remaining buffer space in any other buffer queue, can be used to supplement the buffer space of the video buffer queue. This can expand the buffer space of the video buffer queue to receive more data and prevent the modem's data transmission and retrieval thread from being blocked, allowing the playback to proceed normally without occupying additional system storage space.
[0130] For example, assuming that the maximum buffer data amount allowed by the audio buffer queue is 100M and the maximum buffer data amount allowed by the video buffer queue is 100M, then if the video buffer queue can be supplemented by the remaining buffer space in the audio buffer queue, the available buffer space of the video buffer queue is increased from 100M to 200M.
[0131] Furthermore, assuming that Figure 1In the player architecture shown, the maximum buffer data volume allowed for the audio buffer queue, the video buffer queue, and each of the three subtitle buffer queues is 100M. If the video buffer queue can be supplemented by the remaining buffer space in any other buffer queue, the available buffer space of the video buffer queue is increased from 100M to 500M.
[0132] It can be seen from the above embodiments that the method provided by the embodiments of the present invention can effectively and dynamically expand the buffer space of the video buffer queue, improve the player performance, and greatly avoid playback jamming during playback.
[0133] For example, Fig.11 As shown in FIG. 1 , it is a schematic diagram of a video data transmission method in another scenario, such as Fig.11 As shown in (a) in the figure, all audio buffer queues and video buffer queues in this scenario are full. At this time, the first video data cannot be stored in the video buffer queue, and the audio buffer queue has no remaining buffer space for the video buffer queue. Fig.11 As shown in (b), at this time, the modem in the player can create a dynamic buffer queue locally and store the first video data into the dynamic buffer queue.
[0134] In the above embodiment, when the allocated buffer queues are full, the dynamic buffer queue can be enabled in the modem, and the new video data is no longer sent to the system buffer queue, but is temporarily buffered locally and placed in a dynamic buffer queue. At the same time, the video data and the audio data are synchronized in the modem. After the video data and the audio data are synchronized to a certain time difference range, the audio and video data are sent normally. This processing method adopts a dynamic buffer strategy, and only applies for additional memory for the next level of buffering when it is really needed. Under normal circumstances, it does not need to occupy system memory resources, which ensures that special film sources will not be stuck within the range allowed by hardware resources, and will not occupy additional system resources under normal circumstances.
[0135] In some embodiments, after the dynamic buffer queue is created and the first video data is stored in the dynamic buffer queue, the modem can continue to demultiplex the received multimedia file and obtain new data. If the new data obtained is the second video data, the second video data can be directly stored in the dynamic buffer queue at this time.
[0136] Since the motivation for creating a dynamic buffer queue is that there is no remaining free buffer space available in the video buffer queue and other buffer queues, if a dynamic buffer queue has been created, no judgment is required and the newly received second video data can be directly stored in the dynamic buffer queue.
[0137] In some embodiments, Fig.10 As shown in the above Figure 8 After the above S806 (creating a dynamic buffer queue), the video data transmission method provided by the embodiment of the present invention may further perform the following steps:
[0138] S808: Calculate the synchronization time difference of the audio and video in real time.
[0139] The synchronization time difference of audio and video can be understood as the synchronization time difference between audio data and video data, specifically the time difference between the display timestamp of the latest video data and the display timestamp of the latest audio data in the video buffer queue, that is, GAP = PTSvideo-PTSaudio. GAP represents the synchronization time difference of audio and video, and PTSvideo represents the display timestamp of the latest video data.
[0140] The latest video data refers to the latest acquired video data, and the latest audio data refers to the latest acquired audio data.
[0141] When all buffer queues are full, a notification message indicating that the buffer queue is full is sent to the modem while the dynamic buffer queue is created. After the modem receives the notification message, the synchronization time difference of the audio and video at this time can be calculated.
[0142] S809: When the synchronization time difference is less than or equal to the preset duration, the data in the dynamic buffer queue is stored in the video buffer queue.
[0143] In an embodiment of the present invention, it can be determined whether the synchronization time difference is less than or equal to a preset duration. When the synchronization time difference is less than or equal to the preset duration, the data in the dynamic buffer queue is stored in the video buffer queue. When the synchronization time difference is greater than the preset duration, it means that the synchronization time difference between the audio data and the video data is large, and the video data is consumed slowly in the video buffer queue, and synchronization cannot be achieved. At this time, the video buffer queue still cannot receive the data in the dynamic buffer queue.
[0144] The preset duration is smaller than the maximum synchronization time difference of the video buffer queue, and the maximum synchronization time difference is the time difference between the timestamp of the earliest video data and the timestamp of the latest video data in the video buffer queue.
[0145] Specifically, the preset duration may be the product of the maximum synchronization time difference and a preset ratio. The preset ratio is not limited in the embodiment of the present invention and may be a value less than 1 such as 80%, 90% or 70%.
[0146] When the data read by the modem begins to contain audio data, the player can start to continue playing audio and video, while consuming the video data in the video buffer queue, so that the GAP value will gradually decrease. When the synchronization time difference is less than or equal to the maximum synchronization time difference and the preset ratio, a certain buffer space will appear in the video buffer queue to receive the video data, and the modem can store the video data in the dynamic buffer queue into the video buffer queue. For example, assuming that the preset ratio is 90%, the synchronization time difference is expressed as: GAP, and the maximum synchronization time difference is expressed as MAX GAP , then when GAP≤90%*MAX GAP When the video buffer queue is used to receive video data, 10% of the buffer space will appear in the video buffer queue to receive video data.
[0147] S810: When all the data in the dynamic buffer queue has been stored in the video buffer queue, cancel the dynamic buffer queue.
[0148] As the video data in the video buffer queue is continuously consumed, when all the data in the dynamic buffer queue has been stored in the video buffer queue and there is no need to store any more data in the video buffer queue, the dynamic buffer queue can be cancelled.
[0149] In some embodiments, when the dynamic buffer queue is cancelled, if the modem receives video data again, the video data is preferentially stored in the video buffer queue. Specifically, this can be implemented according to the video data transmission method shown in S801 to S807 above.
[0150] In the above embodiment, the synchronization status of audio data and video data can be timely known through the synchronization time difference between real-time audio data and video data, thereby obtaining the consumption of buffered data in the video buffer queue. When it is known that the synchronization time difference is less than a certain value, it can be determined that the buffered data in the video buffer queue has been consumed to the point where there is sufficient buffer space to receive new data. At this time, the data in the dynamic buffer queue can be stored in the video buffer queue in a timely manner.
[0151] Furthermore, after all the video data in the dynamic buffer queue have been stored in the video buffer queue, the dynamic buffer queue is cancelled in time, so that system storage resources can be released in time to avoid long-term occupation and waste of resources.
[0152] In some embodiments, Fig.11 As shown, the above Fig.10 The illustrated S808 and S809 may also be replaced by the following steps S808a and S809a:
[0153] S808a: Monitor the amount of buffered data in the video buffer queue in real time.
[0154] In some embodiments, the amount of buffered data in the video buffer queue can be calculated by detecting the remaining capacity of the memory space allocated to the video buffer queue in real time and according to the remaining capacity and the total capacity of the memory space allocated to the video buffer queue.
[0155] S809a: When the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount, the data in the dynamic buffer queue is stored in the video buffer queue.
[0156] Wherein, the target buffer amount is smaller than the first preset buffer amount.
[0157] In some embodiments, the target buffer amount may be the product of the first preset buffer amount and a preset ratio. The preset ratio is not limited in the embodiments of the present invention and may be 80%, 85% or 75% or other values less than 1.
[0158] Since the first preset buffer size is the maximum allowable data buffer size set in advance for the video buffer queue, when the amount of buffered data in the video buffer queue is less than or equal to 80% of the maximum data buffer size, 20% of the buffer space will appear in the video buffer queue to receive video data, and the modem can store the video data in the dynamic buffer queue into the video buffer queue.
[0159] In the above embodiment, the amount of buffered data in the video buffer queue can be monitored in real time to timely know the buffer data consumption situation. When it is known that the buffered data in the video buffer queue has been consumed to a certain proportion of the maximum allowed buffer amount in the video buffer queue, the data in the dynamic buffer queue can be stored in the video buffer queue in time.
[0160] Furthermore, after all the video data in the dynamic buffer queue have been stored in the video buffer queue, the dynamic buffer queue is cancelled in time, so that system storage resources can be released in time to avoid long-term occupation and waste of resources.
[0161] In some embodiments, after the dynamic buffer queue is created, the synchronization time difference of the audio and video can also be calculated in real time, and the amount of buffered data in the video buffer queue can be monitored in real time. When the synchronization time difference is less than or equal to the preset duration, and the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount, the data in the dynamic buffer queue is stored in the video buffer queue. In other words, the amount of buffered data and the synchronization time difference in the video buffer queue can be detected at the same time to determine the data consumption in the video buffer queue. After the amount of data in the video buffer queue is gradually reduced to a certain extent, the data in the dynamic buffer queue is stored in the video buffer queue.
[0162] In the above embodiment, the amount of buffered data in the video buffer queue and the synchronization time difference of the audio and video data can be monitored in real time. The buffered data consumption status can be jointly judged based on the amount of buffered data in the video buffer queue and the synchronization time difference. When it is known that the buffered data in the video buffer queue has been consumed to the point where new data can be received, the data in the dynamic buffer queue can be stored in the video buffer queue in a timely manner.
[0163] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned video data transmission method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0164] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0165] The present invention provides a computer program product, characterized in that it includes: when the computer program product is run on a computer, the computer is enabled to implement the above-mentioned video data transmission method.
[0166] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A display device, characterized in that: include: The controller is configured to: obtain first video data; When the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue and the buffered data in the target buffer queue has not reached a second preset buffer amount of the target buffer queue, setting the first preset buffer amount of the video buffer queue to a third preset buffer amount and setting the second preset buffer amount of the target buffer queue to a fourth preset buffer amount; Storing the first video data in the video buffer queue; Among them, the third preset buffer amount is increased by a preset amount compared to the first preset buffer amount, the fourth preset buffer amount is reduced by the preset amount compared to the second preset buffer amount, and the target buffer queue is any queue in the other buffer queues, and the other buffer queues are used to buffer data other than video data.
2. The display device according to claim 1, characterized in that The controller is further configured to: After acquiring the first video data, when the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue, and the buffered data in the other buffer queues have reached preset buffer amounts of their respective buffer queues, creating a dynamic buffer queue; The first video data is stored in the dynamic buffer queue.
3. The display device according to claim 1, characterized in that The target buffer queue is an audio buffer queue, and the controller is further configured to: After acquiring the first video data, when the buffered data in the video buffer queue has reached the first preset buffer amount of the video buffer queue and the buffered data in the audio buffer queue has reached the second preset buffer amount, creating a dynamic buffer queue; The first video data is stored in the dynamic buffer queue.
4. The display device according to claim 2 or 3, characterized in that: The controller is further configured to: After creating the dynamic buffer queue, obtaining second video data; The second video data is directly stored in the dynamic buffer queue.
5. The display device according to claim 2 or 3, characterized in that: The controller is further configured to: After the dynamic buffer queue is created, the synchronization time difference of the audio and video is calculated in real time; wherein the synchronization time difference is the time difference between the display timestamp of the latest video data and the display timestamp of the latest audio data in the video buffer queue; When the synchronization time difference is less than or equal to a preset duration, storing the data in the dynamic buffer queue into the video buffer queue; The preset duration is smaller than the maximum synchronization time difference of the video buffer queue, and the maximum synchronization time difference is the time difference between the timestamp of the earliest video data and the timestamp of the latest video data in the video buffer queue.
6. The display device according to claim 5, characterized in that The controller is further configured to: When all the data in the dynamic buffer queue has been stored in the video buffer queue, the dynamic buffer queue is cancelled.
7. The display device according to claim 2 or 3, characterized in that: The controller is further configured to: After creating the dynamic buffer queue, monitoring the amount of buffered data in the video buffer queue in real time; When the amount of buffered data in the video buffer queue is less than or equal to the target buffer amount, storing the data in the dynamic buffer queue into the video buffer queue; Wherein, the target buffer amount is smaller than the first preset buffer amount.
8. A video data transmission method, characterized in that: include: Acquire first video data; When the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue and the buffered data in the target buffer queue has not reached a second preset buffer amount of the target buffer queue, setting the first preset buffer amount of the video buffer queue to a third preset buffer amount and setting the second preset buffer amount of the target buffer queue to a fourth preset buffer amount; Storing the first video data in the video buffer queue; Among them, the third preset buffer amount is increased by a preset amount compared to the first preset buffer amount, the fourth preset buffer amount is reduced by the preset amount compared to the second preset buffer amount, and the target buffer queue is any queue in the other buffer queues, and the other buffer queues are used to store data other than video data.
9. The method according to claim 8, characterized in that After acquiring the first video data, the method further includes: When the buffered data in the video buffer queue has reached a first preset buffer amount of the video buffer queue, and the buffered data in the other buffer queues have reached preset buffer amounts of their respective buffer queues, creating a dynamic buffer queue; The first video data is stored in the dynamic buffer queue.
10. The method according to claim 8, characterized in that The target buffer queue is an audio buffer queue. After acquiring the first video data, the method further includes: After acquiring the first video data, when the buffered data in the video buffer queue has reached the first preset buffer amount of the video buffer queue and the buffered data in the audio buffer queue has reached the second preset buffer amount, creating a dynamic buffer queue; The first video data is stored in the dynamic buffer queue.
Citation Information
Patent Citations
Queue-based task flow monitoring method and system
CN105516024A
Method and device for determining video buffering
CN106973321A
