Display device and video processing method

Abstract

The present application provides an embodiment, which belongs to display technology, and provides a display device and a video processing method. The display device includes a player and a display screen; the player is configured to: obtain a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file, and multimedia data, the multimedia data including audio data and / or video data; according to the track information, perform framing processing on the multimedia data to obtain target multimedia data; decode the target multimedia data; and transmit the decoded target multimedia data to the display screen for display. The present application can reduce the system consumption of the player by framing the multimedia data parsed by demultiplexing, and performing subsequent decoding and other processing on the framed target multimedia data, thereby improving the playback fluency and key response performance of the player.

Description

Technical Field

[0001] The present application relates to multimedia processing technology, and more specifically, to a display device and a video processing method. Background Art

[0002] The currently used player includes a file download module, an audio and video decapsulation module, a decoding module, a synchronization module, and an audio and video rendering output module. Among them, the audio and video decapsulation module includes: a demultiplexing unit and an audio selector. The demultiplexing module is used to demultiplex the multimedia files downloaded by the file download module to obtain video data and audio data. Then, the audio selector, the decoding module, the synchronization module, and the audio and video rendering output module process the video data and audio data accordingly, and then multimedia playback can be performed.

[0003] For the above-mentioned player, every time the demultiplexing unit demultiplexes a frame of video data or multiple audio data, the audio selector, decoding module, synchronization module and audio and video rendering output module will perform a corresponding functional processing. Therefore, for multimedia files of the same length and format, the more video frames the demultiplexing unit parses, the higher the working frequency of the audio selector, decoding module, synchronization module and audio and video rendering output module will be, which will lead to greater system consumption of the player, and further cause performance issues such as player playback jamming and slow button response. Summary of the invention

[0004] The exemplary implementation of the present application provides a display device and a video processing method, which can effectively reduce the system consumption of a player and thus improve the playback performance of the player.

[0005] In a first aspect, an embodiment of the present application provides a display device, including: a player and a display screen;

[0006] The player is configured as:

[0007] Acquire a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file and multimedia data, where the multimedia data includes audio data and / or video data;

[0008] According to the track information, the multimedia data is framed to obtain target multimedia data;

[0009] Decoding target multimedia data;

[0010] The decoded target multimedia data is transmitted to the display screen for display.

[0011] In some possible implementations, the multimedia data includes at least one channel of video data, the video data includes a plurality of video frames, and the track information includes video track information;

[0012] When the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, it is specifically used for:

[0013] Determine whether the video data is multi-channel video data according to the video track information;

[0014] If the video data is multi-channel video data, a buffer queue corresponding to each channel of video data in the multi-channel video data is created; and each channel of video data is correspondingly stored in the buffer queue;

[0015] In a plurality of buffer queues, video frames with the same first timestamp are obtained for framing to obtain target video frames.

[0016] In some possible implementations, when the player obtains video frames with the same timestamp from multiple buffer queues and groups them to obtain a target video frame, it is specifically used to:

[0017] Obtain the video frame at the head position of each buffer queue as the head video frame;

[0018] Get the first timestamp of each team head video frame;

[0019] Determine whether the first timestamps are the same. If they are the same, group the head video frames to obtain the target video frames.

[0020] In some possible implementations, the player is used to:

[0021] If the video data is not multi-channel video data, determining a first duration of the video frame according to the video track information;

[0022] When the first duration is less than a first preset duration threshold, determining that each target video frame includes a first number of video frames according to the first duration and the first preset duration threshold;

[0023] Determine a first timestamp of each video frame;

[0024] According to the order corresponding to the first timestamp, the first number of video frames are selected in sequence for framing to obtain the target video frame.

[0025] In some possible implementations, the at least one channel of video data includes: a channel of base layer video data and a channel of enhancement layer video data. Before the player stores the respective channels of video data in the buffer queue, the player is specifically configured to:

[0026] Adding target identification in the enhanced layer video frame;

[0027] When the player decodes the target multimedia data, it is specifically used to:

[0028] In the target video frame, identifying the enhancement layer video frame and the base layer video frame according to the target identifier;

[0029] The base layer video frame is first decoded and processed, and then the enhanced layer video frame is decoded and processed; wherein the base layer video data includes: a plurality of base layer video frames; the enhanced layer video data includes: a plurality of enhanced layer video frames.

[0030] In some possible implementations, the video track information includes: video metadata; when the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, it is specifically used to:

[0031] The video metadata and each video frame are grouped to obtain a plurality of target video frames.

[0032] In some possible implementations, when the player performs framing processing on the multimedia data according to the track information to obtain the target multimedia data, it is specifically used to:

[0033] Grouping video frames of the video data to obtain intermediate video frames;

[0034] The video metadata is combined with each intermediate video frame to obtain a plurality of target video frames.

[0035] In some possible implementations, the multimedia data includes: at least one channel of audio data, the track information includes: audio track information, the audio data includes a plurality of audio frames;

[0036] When the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, it is specifically used for:

[0037] Determine a second duration of the audio frame according to the audio track information;

[0038] If the second duration is less than the second preset duration threshold, the audio frames are grouped for each channel of audio data to obtain a plurality of target audio frames.

[0039] In some possible implementations, when the player groups audio frames for each channel of audio data to obtain multiple target audio frames, it is specifically used to:

[0040] Determine, according to the second duration and the second preset duration threshold, that each target audio frame contains a second number of audio frames;

[0041] determining a second timestamp of the audio frame;

[0042] According to the order corresponding to the second timestamp, the second number of audio frames are selected in sequence for framing to obtain the target audio frame.

[0043] In a second aspect, an embodiment of the present application provides a video processing method, which is applied to a display device. The video processing method includes:

[0044] Acquire a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file and multimedia data, where the multimedia data includes audio data and / or video data;

[0045] According to the track information, the multimedia data is framed to obtain target multimedia data;

[0046] Decoding target multimedia data;

[0047] The decoded target multimedia data is transmitted to the display screen for display.

[0048] In a third aspect, an embodiment of the present application provides a video processing device, which is applied to a display device. The video processing device includes:

[0049] A demultiplexing module, used to obtain multimedia files and demultiplex the multimedia files to obtain track information of the multimedia files and multimedia data, where the multimedia data includes audio data and / or video data;

[0050] The framing module is used to perform framing processing on the multimedia data according to the track information to obtain the target multimedia data; the decoding module is used to decode the target multimedia data;

[0051] The transmission module is used to transmit the decoded target multimedia data to the display screen for display.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed, any video processing method of the second aspect of the present application is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any touch video processing method as described in the second aspect of the present application.

[0054] The display device, video processing method and apparatus provided by the present application, the display device includes a player and a display screen; the display device includes a player and a display screen; the player is configured to: obtain a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file, and multimedia data, the multimedia data including audio data and / or video data; according to the track information, perform framing processing on the multimedia data to obtain target multimedia data; decode the target multimedia data; transmit the decoded target multimedia data to the display screen for display. The present application can reduce the system consumption of the player by framing the multimedia data parsed by demultiplexing, and performing subsequent decoding and other processing on the framed target multimedia data, thereby improving the playback fluency and key response performance of the player. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0056] Figure 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a user according to an embodiment;

[0057] Figure 2 exemplarily shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment;

[0058] Figure 3 A structural block diagram of a player provided in an embodiment of the present application;

[0059] Figure 4 A flowchart of a video processing method provided by an embodiment of the present application;

[0060] Figure 5 A structural block diagram of a player provided in another embodiment of the present application;

[0061] Figure 6 A flowchart of a video processing method provided by another embodiment of the present application;

[0062] Figure 7 A structural block diagram of a player provided in yet another embodiment of the present application;

[0063] Figure 8 A flowchart of a video processing method provided by another embodiment of the present application;

[0064] Fig. 9 A schematic diagram of video data composition provided in an embodiment of the present application;

[0065] Fig.10 A flowchart of a video processing method provided in yet another embodiment of the present application;

[0066] Fig.11 A schematic diagram of the structure of a video processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0068] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0069] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0070] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to orders other than those given in the diagrams or descriptions of the embodiments of this application.

[0071] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0072] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0073] Figure 1FIG. 2 exemplarily shows a schematic diagram of an operation scenario between a display device and a user according to an embodiment. Figure 1 As shown in FIG. 1 , a user can perform writing operations on the capacitive touch screen of the display device 200 by using the touch pen 100 , and the processor of the display device 200 determines the touch point according to the touch operation on the capacitive touch screen.

[0074] like Figure 1 It is also shown that the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. For example, the display device 200 receives software program updates or accesses a remotely stored digital media library by sending and receiving information and interacting with an electronic program guide (EPG). The server 400 may be a cluster or multiple clusters, and may include one or more types of servers. Other network service content such as video on demand and advertising services is provided through the server 400.

[0075] The embodiment of the present application does not limit the type, size, resolution, etc. of the specific display device 200. Those skilled in the art will appreciate that the display device 200 may be modified in terms of performance and configuration as needed.

[0076] Figure 2 Schematically shows a hardware configuration block diagram of a display device 200 according to an exemplary embodiment.

[0077] In some embodiments, the display device 200 includes at least one of a controller 250, a tuner and demodulator 210, a communicator 220, a detector 230, an input / output interface 255, a display 275, an audio output interface 285, a memory 260, a power supply 290, a user interface 265, and an external device interface 240.

[0078] In some embodiments, the display 275 is a component for receiving the image signal outputted from the first processor and displaying video content and images as well as a menu control interface.

[0079] In some embodiments, the display 275 includes a display screen component for presenting images and a driving component for driving image display.

[0080] In some embodiments, the video content is displayed, and various image contents sent from a network server end can be displayed via a network communication protocol.

[0081] In some embodiments, the display 275 is used to present a user manipulation UI interface generated in the display device 200 and used to control the display device 200 .

[0082] In some embodiments, depending on the type of display 275, a driving component for driving the display is also included.

[0083] In some embodiments, the communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example, the communicator may include at least one of a Wifi chip, a Bluetooth communication protocol chip, a wired Ethernet communication protocol chip, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

[0084] In some embodiments, the display device 200 may establish control signal and data signal transmission and reception with an external control device or a content providing device through the communicator 220 .

[0085] In some embodiments, the user interface 265 may be used to receive infrared control signals from a control device (eg, an infrared remote controller, etc.).

[0086] In some embodiments, the detector 230 is a signal used by the display device 200 to collect external environment or interact with the outside.

[0087] In some embodiments, the detector 230 includes a light receiver, a sensor for collecting ambient light intensity, and can adaptively display parameter changes by collecting ambient light.

[0088] In some embodiments, the detector 230 may also include a temperature sensor or the like, such as by sensing the ambient temperature.

[0089] In some embodiments, the display device 200 can adaptively adjust the display color temperature of the image. For example, when the temperature is high, the display device 200 can be adjusted to display a cooler color temperature of the image, or when the temperature is low, the display device 200 can be adjusted to display a warmer color temperature of the image.

[0090] In some embodiments, the detector 230 may also be a sound collector, such as a microphone, which can be used to receive the user's voice, for example, including a voice signal of a control instruction of the user to control the display device 200, or collect environmental sounds to identify the type of environmental scene, so that the display device 200 can adaptively adapt to the environmental noise.

[0091] In some embodiments, Figure 2 As shown, the input / output interface 255 is configured to perform data transmission between the controller 250 and other external devices or other controllers 250, such as receiving video signal data and audio signal data, or command instruction data from external devices.

[0092] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more interfaces such as a high-definition multimedia interface HDMI interface, an analog or digital high-definition component input interface, a composite video input interface, a USB input interface, an RGB port, etc. It is also possible that the above multiple interfaces form a composite input / output interface.

[0093] In some embodiments, Figure 2 As shown, the tuner-demodulator 210 is configured to receive broadcast television signals through wired or wireless reception, and can perform modulation and demodulation processing such as amplification, mixing and resonance, and demodulate audio and video signals from multiple wireless or wired broadcast television signals. The audio and video signals may include television audio and video signals carried in the television channel frequency selected by the user, as well as EPG data signals.

[0094] In some embodiments, the frequency demodulated by the tuner-demodulator 210 is controlled by the controller 250, which can send a control signal based on the user's selection to make the modem respond to the TV signal frequency selected by the user and modulate and demodulate the TV signal carried by the frequency.

[0095] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 can control the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 275, the controller 250 can perform operations related to the object selected by the user command.

[0096] In some embodiments, the object may be any one of the selectable objects, such as a hyperlink or an icon. Operations related to the selected object may include, for example, displaying operations such as a page, document, or image connected to the hyperlink, or executing operations corresponding to a program of the icon. User commands for selecting UI objects may be commands inputted through various input devices (e.g., a mouse, a keyboard, a touchpad, etc.) connected to the display device 200, or voice commands corresponding to voices spoken by the user.

[0097] like Figure 2As shown, the controller 250 includes at least one of a random access memory 251 (Random Access Memory, RAM), a read-only memory 252 (Read-Only Memory, ROM), a video processor 270, an audio processor 280, other processors 253 (for example: a graphics processor (Graphics Processing Unit, GPU), a processor 254 (Central Processing Unit, CPU), a communication interface (Communication Interface), and a communication bus 256 (Bus). Among them, the communication bus connects various components.

[0098] Reference Figure 3 A schematic diagram of the structure of a player is shown, including a download module 11, a decapsulation module 12, an audio decoding module 13, a video decoding module 14, an audio and video synchronization module 15, an audio output module 16, and a video output module 17. The decapsulation module 12 includes: a demultiplexing unit 121, at least one audio elementary stream queue 122, a video elementary stream queue 123, and an audio selection unit 124.

[0099] Specifically, the download module 11 is used to download multimedia files from the server. The demultiplexing unit 121 is used to demultiplex the multimedia files and demultiplex the video data and / or audio data. The audio elementary stream queue 122 is used to store the demultiplexed multi-channel audio data, wherein one channel of audio data is correspondingly stored in one audio elementary stream queue 122. The video elementary stream queue 123 is used to store the demultiplexed video data. The audio selection unit 124 is used to select the multi-channel audio data. The audio decoding module 13 is used to decode the selected audio data. The video decoding module 14 is used to decode the video data in the video elementary stream queue 123. The audio and video synchronization module 15 is used to synchronize the decoded video data with the audio data. The audio output module 16 is used to render and output the audio data. The video output module 17 is used to render and output the video data. Finally, it is transmitted to the display screen and displayed on the display screen.

[0100] Furthermore, the player needs to play multiple types of multimedia files, such as: MVC (multi-view video), Dolby Vision, HLG (Hybrid Log Gamma), etc. Different types of multimedia files use different video encoding formats and audio encoding formats; among them, the video encoding formats include: H.264, H.265, AV1, VP8, VP9, ​​etc., and the audio encoding formats include: aac, ac3, vorbis. In addition, the number of channels of video data encapsulated in different types of multimedia files can be one or more channels, and the number of channels of audio data can also be one or more channels. And the duration of the video frames in the video data can also be the same or different.

[0101] Specifically, multimedia data includes multiple forms. The first form is: multiple channels of video data and one channel of audio data; the second form is: multiple channels of video data and multiple channels of audio data; the third form is: one channel of video data and one channel of audio data; the fourth form is: one channel of video data and multiple channels of audio data; the fifth form is: only one channel or multiple channels of video data; the sixth form is: only one channel or multiple channels of audio data.

[0102] For the above multimedia data, if Figure 3 The player will have the following problems:

[0103] 1. When there are multiple channels of video data, for example, there are two channels of video data, and the video data includes multiple video frames, the demultiplexing unit 121 will first multiplex one or more video frames of the first channel of video data, store them in the video elementary stream queue 123, and then demultiplex one or more video frames of the second channel of video data, store them in the video elementary stream queue 123. Because the demultiplexing unit 121 does not demultiplex the two channels of video data in the order of timestamps when demultiplexing the two channels of video data, for example, the storage order in the video elementary stream queue 123 may be, 0.1 second to 0.8 second video frames of the first channel of video data, then 0.1 second to 0.4 second video frames of the second channel of video data, and then 0.9 second to 0.9 second video frames of the second channel of video data. The video frames of the first video data of 0.5 seconds to 1.5 seconds are stored, and the video frames of the second video data of 0.5 seconds to 0.8 seconds are stored; when the subsequent video decoding module 14 decodes, it will be decoded according to the storage order of the video basic stream queue 123, which will cause the video frames with the same time stamp of the two video data to be unable to be played at the same time, and ultimately cause abnormal display problems such as flowery screen, half screen or mosaic on the display screen, and each video frame in the video basic stream queue 12 requires the video decoding module 14, the audio and video synchronization module 15 and the video output module 17 to perform a corresponding function once, which will cause the player system to consume a lot, and the playback will be stuck and the key response will be slow.

[0104] 2. When there is one channel of video data and the duration of the video frame in the video data is short, for example, when the duration of the video frame is 0.02 seconds, the video decoding module 14 needs to decode 50 times in 1 second, and the audio and video synchronization module 15 and the video output module 17 also need to execute the corresponding functions 50 times in 1 second, which will lead to high consumption of the player system, playback jams and slow button response.

[0105] 3. When there is one or more channels of audio data, they are all one channel of audio data after passing through the audio selection unit 124. However, when the duration of an audio frame in the audio data is short, for example, when the duration of the audio frame is 0.02 seconds, the audio decoding module 13 needs to decode 50 times in 1 second, and the audio and video synchronization module 15 and the audio output module 16 also need to perform corresponding functions 50 times in 1 second, which will lead to high consumption of the player system, playback jams, and slow button response.

[0106] 4. For multimedia files of MVC and Dolby Vision type, there are two video data channels, one for basic layer video data and the other for enhanced layer video data. In order to make multimedia files of MVC and Dolby Vision type display normal effects on the display screen, the video decoding unit 14 needs to parse out a basic layer video frame first, then parse out an enhanced layer video frame, and then put a basic layer video frame in front and an enhanced layer video frame in the back, and send them to the audio and video synchronization module 15 after combining them, and finally play them. However, using Figure 3 In a player, the video decoding module 14 cannot stably and alternately parse the basic layer video frames and the enhanced layer video frames. When an enhanced layer video frame is parsed first and then a basic layer video frame is parsed, or two identical enhanced layer video frames or basic layer video frames are parsed continuously, frame assembly failure will occur, which will lead to playback problems.

[0107] 5. Usually, the video metadata is placed in the first frame of the video data. The video metadata is used to configure the video decoding module 14 according to the video metadata when the video decoding module 14 parses the video metadata, and decode the subsequent video frames. However, when the playback is resumed, that is, the user closes the last video playback, and the current playback needs to start at the time when the video playback was closed last time, the video decoding module 14 will skip the first frame of the video frame and start decoding directly from the video frame of the resumed playback time. The video decoding module 14 cannot obtain the video metadata, and will not configure the video decoding module 14, which will lead to the problem that the ideal display effect cannot be achieved in the case of resumed playback.

[0108] Based on the above problems, the present application provides a display device, a video processing method and an apparatus, which performs frame processing after demultiplexing the multimedia files, thereby reducing the system consumption of the player and improving the playback performance of the display device.

[0109] The following detailed examples are used to illustrate how this application is carried out.

[0110] Figure 4 The flowchart of the multimedia file processing method provided in one embodiment of the present application is applied to a display device 200, and the display device 200 includes: a player and a display screen. Figure 4 As shown, the player in the display device 200 is configured to perform the following steps:

[0111] In S101, a multimedia file is acquired and demultiplexed to obtain track information and multimedia data of the multimedia file.

[0112] The multimedia file is an audio and video file, or a video file or an audio file. In addition, the multimedia data includes audio data and / or video data.

[0113] Reference Figure 5 , is a structural block diagram of a player provided by the present application, wherein the download module 11 downloads the multimedia file. The demultiplexing unit 121 demultiplexes the multimedia file to obtain track information and multimedia data.

[0114] Specifically, the track information is usually stored in the first video frame of the multimedia data. The track information includes: track encoding format, encapsulated frame sequence number, encoding metadata, track identification information, etc. Among them, the track encoding format includes: video encoding format and audio encoding format. Video encoding formats include: H.264, H.265, AV1, VP8, VP9, ​​etc., and audio encoding formats include aac, ac3, vorbis, etc. The encapsulated frame sequence number identifies the number of tracks encapsulated in the multimedia file. One track has a frame sequence number, and one track corresponds to one channel of video data or one channel of audio data. The encoding metadata includes: video metadata and audio metadata. Video metadata is used to configure the video decoding module 14, and audio metadata is used to configure the audio decoding module 13. In addition, video metadata and audio metadata are both parameters. Exemplarily, the encoding metadata of the video encoding formats H.264 and H.265 are sequence parameter set information and image parameter set information. The track identification information is used to determine the number of channels of video data and the number of channels of audio data included in the multimedia file, wherein the track identification information is encapsulated in the multimedia file. For example, the track box in the MP4 file represents the track identification information, and the pmt in the ts file represents the track identification information.

[0115] Furthermore, after the multimedia file is demultiplexed, one or more channels of video data and one or more channels of audio data are obtained, where multiple channels means greater than or equal to 2 channels.

[0116] In an embodiment of the present application, any type of multimedia file can be demultiplexed to obtain corresponding track information and multimedia data.

[0117] In S102, the multimedia data is framed according to the track information to obtain target multimedia data.

[0118] Among them, some information of multimedia data can be determined from the track information, for example, the multimedia data includes: several channels of video data and several channels of audio data; the duration of video frames in the video data, and the duration of audio frames in the audio data; video metadata, audio metadata, etc.

[0119] In addition, the framing process for multimedia data specifically includes performing different framing processes according to different track information.

[0120] Exemplarily, when the multimedia data includes: multiple channels of video data, a buffer queue 126 is created after the demultiplexing unit 121, and the multiple channels of video data demultiplexed by the demultiplexing unit 121 are stored in the buffer queue 126 accordingly. Video frames with the same timestamp are obtained in each buffer queue 126, and the video frames with the same timestamp can be grouped and finally played simultaneously, thereby avoiding disorder and the display screen showing a flowery screen, half screen or mosaic.

[0121] When the multimedia data includes: one channel of video data, the duration of the video frames in the video data is determined. If the duration of the video frames in the video data is less than the first preset threshold, framing is performed. This can reduce the frequency of processing video frames per unit time by the video decoding module 14, the audio and video synchronization module 15 and the video output module 17, thereby reducing the system consumption of the player.

[0122] When the multimedia data includes: one or more channels of audio data, the length of the audio frame in each channel of audio data is determined. If the duration of the audio frame is less than the second preset threshold, framing is performed. This can reduce the frequency of processing video frames per unit time by the audio decoding module 13, the audio and video synchronization module 15 and the audio output module 16, thereby reducing the system consumption of the player.

[0123] In an embodiment of the present application, the grouped video frames and / or audio frames are used as target multimedia data for subsequent decoding, synchronization and output processing.

[0124] In some embodiments, during the replay process, the first video frame received by the video decoding module 14 is the video frame after the jump, not the first video frame in the multimedia file. When the demultiplexing unit 121 makes a jump, it obtains the key frame data at the jump position, which causes the video metadata stored before the key frame to be unable to be demultiplexed and transmitted to the video decoding module 14; for example, the SEI information (video metadata) before the key frame position in the MP4 file, which causes the video decoding module 14 to be unable to configure according to the video metadata, thereby causing the decoded video to be distorted.

[0125] In some embodiments, based on the above, the video track information includes: video metadata; when the player executes S102, it includes: framing the video metadata and each video frame to obtain multiple target video frames.

[0126] Specifically, in a multimedia file, the video metadata is stored in the first video frame. When the video decoding module 14 decodes the video metadata, the video decoding module 14 can be configured according to the video metadata, and then the subsequent video frames can be decoded according to the configured video decoding module 14.

[0127] Among them, when the multimedia data includes single-channel video data; when the user closed the player last time, and currently needs to start the player from the time when it was last closed to start playing the multimedia content, then because the video frame in the middle position does not contain video metadata, the video decoding module 14 cannot be configured according to the video metadata, which will cause the subsequent decoded video frames to appear in a video that does not conform to the multimedia file format; in the present application, the video metadata and each video frame are grouped together, so that at any time of continued playback, the video decoding module 14 can obtain the video metadata for configuration, so that the decoded video will not be distorted.

[0128] Furthermore, when receiving a resume request, the first video frame corresponding to the resume request is obtained, and the video metadata and the first video frame are grouped together. The video frames subsequent to the first video frame do not need to be grouped together, so that the resume can be achieved without distortion.

[0129] Exemplarily, the multimedia file is an HLG file, and the video metadata in the HLG file is SEI. If the video metadata can be extracted during demultiplexing, the video metadata can be framed with each video frame, or the first video frame corresponding to the resume request. Among them, the video metadata needs to be converted into a format consistent with the video frame. For example, the format of the video frame of the H265 file is HVCC or annexb byte stream, then the video metadata must also be converted into the format of HVCC or annexb byte stream when framing.

[0130] In some embodiments, when the player executes S102, the steps include: grouping video frames of the video data to obtain intermediate video frames; and grouping video metadata with each intermediate video frame to obtain multiple target video frames.

[0131] Specifically, framing the video frames of the video data to obtain the intermediate video frame can specifically be the contents of the following steps S202-S209, and the intermediate video frame is the video frame after the multi-channel video data or the single-channel video data is framed. Exemplarily, when the multimedia data includes: multiple channels of video data, a buffer queue 126 is created after the demultiplexing unit 121, and the multiple channels of video data demultiplexed by the demultiplexing unit 121 are correspondingly stored in the buffer queue 126. Video frames with the same timestamp are obtained in each buffer queue 126, and the video frames with the same timestamp can be framed to obtain the intermediate video frame. When the multimedia data includes: one channel of video data, the duration of the video frames in the video data is determined. If the duration of the video frames in the video data is less than the first preset threshold, framing is performed to obtain the intermediate video frame.

[0132] Further, the video metadata is grouped with each intermediate video frame to obtain multiple target video frames, specifically: the video metadata is inserted into each intermediate video frame to obtain the target video frame; in this way, in the case of a resumed broadcast, the video decoding module 14 can decode the video metadata and configure the video decoding module to decode subsequent target video frames. In addition, when a resumed broadcast request is received, the video metadata is grouped with the intermediate video frames corresponding to the resumed broadcast request.

[0133] In addition, when the multimedia file does not include video metadata, it is not necessary to perform a frame processing on the video frame or the intermediate video frame and the video metadata. Then the intermediate video frame is the target video frame sent to the video decoding module for decoding.

[0134] In an embodiment of the present application, by first grouping the video frames to obtain intermediate video frames, and then grouping the intermediate video frames and video metadata to obtain target video frames, multi-channel video data or single-channel video data can be played without screen distortion, half-screen, mosaic, and freeze. Furthermore, video distortion will not occur during continued playback.

[0135] In S103, the target multimedia data is decoded.

[0136] Specifically, decoding the multimedia data includes: performing audio decoding on the multimedia data and / or performing video decoding on the multimedia data. The audio decoding module 13 is used to perform audio decoding on the audio data in the multimedia data, and the video decoding module 14 is used to perform video decoding on the video data in the multimedia data.

[0137] Furthermore, when the multimedia file includes multiple channels of audio data, the audio selection unit 124 is used to perform selective frame loss processing on the multiple channels of audio data to obtain one channel of audio data, and then the audio decoding module 13 decodes the one channel of audio data.

[0138] In S104, the decoded target multimedia data is transmitted to a display screen for display.

[0139] Specifically, when the target multimedia data includes: target audio data and target video data; the decoded target multimedia data is first transmitted to the audio and video synchronization module for synchronization processing, and then the target audio data is transmitted to the audio output module for audio rendering and output, and the target video data is transmitted to the video output module for video rendering and output, and finally transmitted to the display screen for display.

[0140] The video processing method provided by the present application includes: obtaining a multimedia file, and demultiplexing the multimedia file to obtain track information of the multimedia file, and multimedia data, the multimedia data including audio data and / or video data; framing the multimedia data according to the track information to obtain target multimedia data; decoding the target multimedia data; transmitting the decoded target multimedia data to a display screen for display. The present application can reduce the system consumption of the player by framing the multimedia data parsed by demultiplexing, and performing subsequent decoding and other processing on the framed target multimedia data, thereby improving the playback fluency and key response performance of the player.

[0141] The video processing method provided in the embodiment of the present application is described in detail below in combination with specific steps. Figure 6 This is a flow chart of a video processing method provided by another embodiment of the present application. Figure 6 As shown, the player in the display device 200 is configured to perform the following steps:

[0142] In S201, a multimedia file is acquired and demultiplexed to obtain track information of the multimedia file and multimedia data.

[0143] The specific implementation process of this step can refer to the relevant description of S101 and will not be repeated here. It should be supplemented that, in this embodiment, the multimedia data includes: at least one channel of video data, and may also include audio data.

[0144] In S202, it is determined whether the video data is multi-channel video data according to the video track information.

[0145] Specifically, the multimedia data includes at least one channel of video data, the video data includes multiple video frames, the track information includes video track information; and the video track information includes track identification information.

[0146] In addition, the multi-channel video data is greater than or equal to 2-channel video data. Specifically, the number of channels of the video data is determined to be different according to the track identification information in the video track information. For example, for a Dolby Vision multimedia file, it can be determined based on the track identification information in the video track information that 2-channel video data is included. Therefore, it is possible to determine whether the video data is multi-channel video data according to the video track information.

[0147] If the video data is multi-channel video data, execute S203-S205; if the video data is not multi-channel video data, execute S206-S209.

[0148] In the embodiment of the present application, different frame grouping processes are performed according to different numbers of video data, and frame grouping processes can be performed on different types of multimedia data.

[0149] In S203, a buffer queue corresponding to each channel of video data in the multiple channels of video data is created.

[0150] See also Figure 7 When the demultiplexing unit 121 demultiplexes multiple channels of video data, it creates multiple buffer queues; one channel of video data corresponds to one buffer queue.

[0151] Specifically, it can be determined according to the video track information that the multimedia file specifically includes recorded video data, and then the number of buffer queues to be created is determined.

[0152] The video track information includes: track identification information, video track sequence number and video metadata; each channel of video data corresponds to a video sequence number, and the number of channels of video data can be determined according to the number of video sequence numbers.

[0153] In S204, each channel of video data is stored in a corresponding buffer queue.

[0154] In the embodiment of the present application, one channel of video data corresponds to one buffer queue, and one channel of video data is stored in one buffer queue.

[0155] Wherein, each channel of video data has a video track sequence number, and each buffer queue is identified according to the video track sequence number, and a corresponding relationship between the video track sequence number and the buffer queue identifier is established; exemplarily, the multimedia data includes 2 channels of video data, the video track sequence number of one channel of video data is 000001, and the video track sequence number of the other channel of video data is 000002. The identifiers of the buffer queues can be buffer queue 1 and buffer queue 2, then the video track sequence number 000001 is corresponded to buffer queue 1, and the video track sequence number 000002 is corresponded to buffer queue 2, and then the demultiplexed video frame with the video track sequence number 000001 is stored in buffer queue 1, and the demultiplexed video frame with the video track sequence number 000002 is stored in buffer queue 2.

[0156] In addition, the video frame, audio frame or subtitle includes an index number; the index number is used to indicate which track the demultiplexed data belongs to. For example, the index number of the video frame with the video track sequence number of 000001 is 001, and the index number of the video frame with the video track sequence number of 000002 is 002.

[0157] In an embodiment of the present application, the demultiplexing unit demultiplexes the video frames of one channel of video data in the order of timestamps, but the demultiplexing of the video frames of multiple channels of video data is not necessarily performed alternately. For example, the multimedia data has two channels of video data, the video track sequence number corresponding to one channel of video data is 000001, and the video track sequence number corresponding to the other channel of video data is 000002. Then the demultiplexing unit may parse out the video frame A with the video track sequence number of 000001 and the timestamp of 0.1 seconds for the first time, and store the video frame A in the buffer queue 1 by obtaining the index number of the video frame A as 001. The second time, the video frame B with the video track sequence number of 000002 and the timestamp of 0.1 seconds is parsed out, and the video frame B is stored in the buffer queue 2 by obtaining the index number of the video frame B as 002. The third time, the video frame C with the video track sequence number of 000002 and the timestamp of 0.2 seconds is parsed out, and the video frame C is stored in the buffer queue 2 by obtaining the index number of the video frame C as 002. The fourth parsing results in a video frame D with a video track sequence number of 000002 and a timestamp of 0.3 seconds. By obtaining the index number of video frame D as 002, the video frame D is stored in buffer queue 2. The fifth parsing results in a video frame E with a video track sequence number of 000001 and a timestamp of 0.2 seconds. By obtaining the index number of video frame E as 001, the video frame E is stored in buffer queue 1. The sixth parsing results in a video frame F with a video track sequence number of 000002 and a timestamp of 0.4 seconds. By obtaining the index number of video frame F as 002, the video frame F is stored in buffer queue 2. The seventh parsing results in a video frame G with a video track sequence number of 000001 and a timestamp of 0.3 seconds. By obtaining the index number of video frame G as 001, the video frame G is stored in buffer queue 1. The eighth parsing obtains a video frame H with a video track sequence number of 000002 and a timestamp of 0.4 seconds, and the index number of the video frame H is obtained as 002, and the video frame H is stored in the buffer queue 1. Based on this, the demultiplexing unit 121 can demultiplex the video data of the multimedia file, and then store the demultiplexed video frames in the corresponding buffer queue according to the index number.

[0158] In the embodiment of the present application, by creating a buffer queue, the out-of-order video frames demultiplexed by the demultiplexing unit can be arranged in order in each buffer queue. Specifically, each video frame is stored in the buffer queue by a tail insertion method.

[0159] In S205 , video frames with the same first timestamp are obtained from multiple buffer queues for framing to obtain target video frames.

[0160] Specifically, each video frame has a first timestamp. Among them, multi-channel video data generally refers to video data shot by multiple cameras at the same time, and the video data shot by each camera is one channel of video data. Each video frame in one channel of video data should have a video frame with the same first timestamp in another channel of video data. Therefore, in the present application, video frames with the same first timestamp in multiple buffer queues are grouped, so that when the subsequent video decoding module decodes, the video frames with the same first timestamp can be decoded at one time, avoiding the problem of distorted screen or half-screen display.

[0161] Further, S205 includes: obtaining the video frame at the head position of each buffer queue as the head video frame; obtaining the first timestamp of each head video frame; determining whether the first timestamps are the same, and if they are the same, grouping the head video frames to obtain the target video frame.

[0162] When the video frames are stored in the buffer queue, they are stored according to the tail insertion method. For example, the first video frame of the first video data is demultiplexed first, and the first video frame is stored in the buffer queue. Then, the second video frame of the first video data is demultiplexed, and the second video frame is stored in the buffer queue at a position behind the first video frame. Specifically, the head position of the buffer queue refers to the first position in the buffer queue. When the video frames in the buffer queue are subsequently decoded, the video frame at the head position is first obtained. After the video frame at the head position is read, the subsequent video frames are stored in the head position of the buffer queue.

[0163] Based on the above, if the first timestamps of the video frames at the head positions of the multiple buffer queues are the same, the video frames at the head positions are grouped. In addition, if the first timestamps of the video frames at the head positions of the buffer queues are different, the video frames with smaller first timestamps are subjected to fault tolerance processing, and then the video frames at the head positions of the buffer queues with the same first timestamps are continued to be grouped. Among them, the fault tolerance processing includes: dropping the frame or transmitting the frame to the video decoding module for decoding processing.

[0164] In an embodiment of the present application, by creating a buffer queue, storing video frames, and then obtaining video frames with the same first timestamp at the head of the buffer queue for framing, video frames with the same first timestamp of multiple video data can be quickly and efficiently framed, and the video decoding module can decode the video frames in an orderly manner, thereby improving the quality of the images displayed on the display screen.

[0165] In S206, a first duration of the video frame is determined according to the video track information.

[0166] In an embodiment of the present application, based on the video track information, it can be determined that the video data is not multi-channel video data, that is, it is one channel of video data, and when it is one channel of video data, the first duration of each video frame can be determined based on the video track information.

[0167] Specifically, usually, when it is one channel of video data, since the duration of the video frame is relatively large, it will not increase the consumption of the player system, so frame grouping is not required. And when the video data includes key frames and other video frames, since the first durations of the key frames and other video frames are different, they cannot be grouped at will. However, when all video frames are key frames, since the first duration of the key frames is relatively small, it will increase the system consumption of the player, and the first durations of the key frames are the same, frame grouping is required and can be performed.

[0168] Exemplarily, when the video track information carries the tag data of EXT-XI-FRAMES-ONLY, it can be determined that all the video frames in the video data are key frames.

[0169] In the embodiment of the present application, the video track information can represent the relevant parameters of the video frame during video encoding, such as the first duration of the video frame, timestamp, etc. In addition, the first duration of each video frame is the same.

[0170] In addition, before S206, it also includes: determining whether the video data includes a combined video frame. If it includes a combined frame, S206 includes: determining a first duration of the combined video frame, and when the first duration is less than a first preset duration threshold, executing step S207 to perform frame grouping again; when the first duration is greater than or equal to the first preset duration threshold, exiting frame grouping and directly transmitting the combined video frame to the video decoding module for decoding.

[0171] In S207, when the first duration is less than a first preset duration threshold, it is determined that each target video frame includes a first number of video frames according to the first duration and the first preset duration threshold.

[0172] Specifically, the first preset duration threshold is pre-set. The integer obtained by dividing the first preset duration threshold by the first duration can be used as the first number. For example, when the video decoding module is required to start decoding once in 1 second, the first preset duration threshold is 1 second. When the video decoding module is required to start decoding twice in 1 second, the first preset duration threshold is 0.5 seconds. When the first preset duration threshold is 1 second and the first duration of the video frame is 0.1 seconds, the first number is 10.

[0173] In S208 , a first timestamp of each video frame is determined.

[0174] Specifically, each video frame in the video data corresponds to a first timestamp, and the first timestamp represents the display order of the video frames on the display screen. For example, the first timestamps of the video frames in the video data are 0.1 seconds, 0.2 seconds, 0.3 seconds, etc.

[0175] In S209, according to the order corresponding to the first timestamp, the first number of video frames are selected in sequence for framing to obtain the target video frame.

[0176] Specifically, according to the order from small to large corresponding to the first timestamp, the first number of video frames is selected for framing. For example, video frames with a first timestamp of 0.1 seconds to 1 second are selected for framing, video frames with a first timestamp of 1.1 seconds to 2 seconds are selected for framing, and video frames with a first timestamp of 2.1 seconds to 3 seconds are selected for framing, and subsequent framing processing is performed in this way. The 10 framed video frames are used as a target video frame and transmitted to the video decoding module for decoding. In this way, the number of decoding times of the video decoding module is one tenth of that when no framing is performed, and the corresponding audio and video synchronization module and video output module are also executed one tenth of that when no framing is performed, which greatly reduces the system consumption of the player.

[0177] In S210, the target multimedia data is decoded.

[0178] The specific implementation process of this step can refer to the relevant description of S103, which will not be repeated here.

[0179] In S211, the decoded target multimedia data is transmitted to a display screen for display.

[0180] The specific implementation process of this step can refer to the relevant description of S104, which will not be repeated here.

[0181] In the embodiment of the present application, when the video data is multi-channel video data, the video frames with the same first timestamp in each channel of video data can be grouped and then transmitted to the video decoding module for decoding, thereby achieving orderly decoding of the video frames by the video decoding module, thereby improving the quality of video display. When the video data is one channel of video data, when the first duration of the video frame is less than the first preset threshold, multiple video frames can be grouped and then transmitted to the video decoding module for decoding, which can reduce the system consumption of the player and improve the smoothness of playback.

[0182] Figure 8 FIG. 1 is a flowchart of a video processing method provided by another embodiment of the present application. Figure 8 As shown, the player in the display device 200 is configured to perform the following steps:

[0183] In S301, a multimedia file is acquired and demultiplexed to obtain track information of the multimedia file and multimedia data.

[0184] The specific implementation process of this step can be found in the relevant description of S101, which will not be repeated here. It should be noted that when the multimedia file is MVC or Dolby Vision, the multimedia file includes two channels of video data, one of which is basic layer video data and the other is enhanced layer video data. Only when the video decoding module first decodes a basic layer video frame and then decodes an enhanced layer video frame can the normal playback of the MVC or Dolby Vision multimedia file be achieved.

[0185] In S302, it is determined whether the video data is multi-channel video data according to the video track information.

[0186] The specific implementation process of this step can refer to the relevant description of S202, which will not be repeated here. It should be additionally explained that, according to the video track information, the video data is determined to be two-channel video data, one of which is the basic layer video data and the other is the enhanced layer video data.

[0187] In S303, a buffer queue corresponding to each channel of video data in the multiple channels of video data is created.

[0188] The specific implementation process of this step can refer to the relevant description of S203, which will not be repeated here. It should be supplemented that, here, two buffer queues are created, one buffer queue corresponding to the basic layer video data, and the other buffer queue corresponding to the enhancement layer video data.

[0189] In S304, a target identifier is added to the enhancement layer video frame.

[0190] Specifically, the target identifier is used to represent the enhancement layer video frame, and each enhancement layer video frame includes the target identifier.

[0191] For example, see Fig. 9 , a method for adding a target identifier to Dolby Vision enhancement layer video data. The enhancement layer video data includes multiple enhancement layer video frames, and an enhancement layer video frame is composed of an enhancement layer access unit and a video metadata. An enhancement layer access unit includes: multiple NALUs (Network abstract layer unit). A NALU includes a NAL (Network Abstract Layer) start code, a NAL header and a NAL payload. The video metadata is specifically a byte stream with a NAL unit type of 39 and a SEI type of 147.

[0192] The target identifier is added between the NAL start code and the NAL header. The target identifier may be an identifier byte, for example, the target identifier may be set to TE01 in Dolby Vision.

[0193] In S305, each channel of video data is stored in a corresponding buffer queue.

[0194] The specific implementation process of this step can be referred to the relevant description of S204, which will not be repeated here. It should be additionally explained that here the basic layer video data is stored in one buffer queue, and the other channel of video data with the target identifier added is stored in another buffer queue.

[0195] In S306, video frames with the same first timestamp are obtained from multiple buffer queues for framing to obtain target video frames.

[0196] The specific implementation process of this step can be referred to the relevant description of S205 and will not be repeated here. It should be additionally explained that, here, a basic layer video frame and an enhanced layer video frame with a target identifier added are combined into a frame.

[0197] In S307, in the target video frame, the enhancement layer video frame and the base layer video frame are identified according to the target identifier.

[0198] Specifically, the assembled target video frame includes: a base layer video frame and an enhanced layer video frame with a target identifier added; when the video decoding module performs decoding, the enhanced layer video frame and the base layer video frame in the target video frame are first identified.

[0199] In S308, the base layer video frame is first decoded, and then the enhancement layer video frame is decoded.

[0200] The basic layer video data includes: a plurality of basic layer video frames; the enhanced layer video data includes: a plurality of enhanced layer video frames.

[0201] Specifically, refer to Fig. 9 , a target video frame includes: a base layer video frame and an enhancement layer video frame. The video decoding module first decodes the base layer video frame, and then decodes the enhancement layer video frame, so that the image displayed on the final display screen meets the requirements of Dolby Vision.

[0202] In S309, the decoded target multimedia data is transmitted to a display screen for display.

[0203] The specific implementation process of this step can refer to the relevant description of S104, which will not be repeated here.

[0204] In the embodiment of the present application, it is aimed at processing multimedia files such as MVC or Dolby Vision, which include one base layer video data and one enhancement layer video data. By adding a target identifier to the enhancement layer video frame, it is possible to identify a base layer video frame and an enhancement layer video frame in a target video frame during subsequent decoding. The base layer video frame is decoded first, and then the enhancement layer video frame is decoded, so that the display screen can play the MVC or Dolby Vision multimedia file normally. In addition, the base layer video frame and the enhancement layer video frame are grouped, which can further enable the orderly decoding of multimedia files such as MVC or Dolby Vision, thereby improving the quality of the image displayed on the display screen.

[0205] Fig.10 FIG. 1 is a flowchart of a video processing method provided by another embodiment of the present application. Fig.10 As shown, the player in the display device 200 is configured to perform the following steps:

[0206] In S401, a multimedia file is acquired and demultiplexed to obtain track information of the multimedia file and multimedia data.

[0207] The specific implementation process of this step can refer to the relevant description of S101, which will not be repeated here. It should be supplemented that in the embodiment of the present application, the multimedia data includes: at least one audio data, the track information includes: audio track information, and the audio data includes multiple audio frames.

[0208] In S402, a second duration of the audio frame is determined according to the audio track information.

[0209] Specifically, the audio track information includes the second duration of the audio frame. The second duration of the audio frame can be determined by the audio track information. Usually, the second duration of the audio frame is the same in one channel of video data.

[0210] In S403, if the second duration is less than the second preset duration threshold, the audio frames are grouped for each channel of audio data to obtain a plurality of target audio frames.

[0211] Specifically, after the multimedia file is demultiplexed, one channel of audio data or multiple channels of audio data are demultiplexed; and for multiple channels of audio data, each channel of audio data is framed separately, and after the framing is completed, it is still multiple channels of framed audio data, which are then selected through the audio selection module to obtain one channel of target audio data.

[0212] Exemplarily, the multimedia data includes: one channel of audio data A and one channel of audio data B; wherein each audio frame in the audio data A is framed in the order of the second timestamp to obtain the framed audio data A. Each audio frame in the audio data B is framed in the order of the second timestamp to obtain the framed audio data B. The framed audio data A and the framed audio data B are transmitted to the audio selection module for selection, and finally, the audio selection module selects only one channel of framed audio data A or framed audio data B.

[0213] Further, S403 includes: determining that each target audio frame contains a second number of audio frames based on the second duration and the second preset duration threshold; determining a second timestamp of the audio frame; and selecting the second number of audio frames in sequence according to the order corresponding to the second timestamps for framing to obtain the target audio frame.

[0214] Specifically, the second preset duration threshold is pre-set. The integer obtained by dividing the second preset duration threshold by the second duration can be used as the second number. For example, when the audio decoding module is required to start decoding once in 1 second, the second preset duration threshold is 1 second. When the audio decoding module is required to start decoding twice in 1 second, the second preset duration threshold is 0.5 seconds. When the second preset duration threshold is 0.5 seconds and the second duration of the audio frame is 0.1 seconds, the first number is 5.

[0215] Specifically, each audio frame in the audio data corresponds to a second timestamp, and the second timestamp represents the playback order of the audio frames on the display screen. For example, the second timestamps of the audio frames in the audio data are 0.1 seconds, 0.2 seconds, 0.3 seconds, etc.

[0216] Specifically, in the order from small to large corresponding to the second timestamp, select the second number of audio frames for framing. For example, select the audio frames with the first timestamp of 0.1 seconds to 0.5 seconds for framing, select the audio frames with the first timestamp of 0.6 seconds to 1 second for framing, select the audio frames with the first timestamp of 1.1 seconds to 1.5 seconds for framing, and perform the subsequent framing processing in this way. The 5 framed audio frames are used as a target audio frame and transmitted to the audio decoding module for decoding. In this way, the number of decoding times of the audio decoding module is one-fifth of that when no framing is performed, and the corresponding audio and video synchronization module and audio output module are also executed one-fifth of that when no framing is performed, which greatly reduces the system consumption of the player.

[0217] In S404, the target multimedia data is decoded.

[0218] In the embodiment of the present application, the target audio data is specifically decoded by an audio decoding module.

[0219] In S405, the decoded target multimedia data is transmitted to a display screen for display.

[0220] In the embodiment of the present application, when the display screen displays the video data, the audio data is played through the speaker.

[0221] In the embodiment of the present application, by performing framing processing on the demultiplexed audio data, the number of decoding times of the audio decoding module and the number of execution times of the audio and video synchronization module and the audio output module can be reduced, thereby greatly reducing the system consumption of the player.

[0222] In the embodiment of the present application, the framing process of audio data and the framing process of video data can be performed synchronously, thereby speeding up the processing speed of the player for multimedia data.

[0223] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0224] Fig.11 Schematic diagram of the structure of a video processing device provided in an embodiment of the present application. The present application embodiment provides a video processing device, which is applied to a display device. Fig.11 As shown, the video processing device 500 includes:

[0225] The demultiplexing module 501 is used to obtain multimedia files and demultiplex the multimedia files to obtain track information of the multimedia files and multimedia data, where the multimedia data includes audio data and / or video data;

[0226] The framing module 502 is used to perform framing processing on the multimedia data according to the track information to obtain the target multimedia data; the decoding module 503 is used to decode the target multimedia data;

[0227] The transmission module 504 is used to transmit the decoded target multimedia data to the display screen for display.

[0228] In some possible implementations, the multimedia data includes at least one channel of video data, the video data includes a plurality of video frames, and the track information includes video track information;

[0229] The framing module 502 is specifically used for: determining whether the video data is multi-channel video data according to the video track information; if the video data is multi-channel video data, creating a buffer queue corresponding to each channel of video data in the multi-channel video data; storing each channel of video data in the buffer queue; obtaining video frames with the same first timestamp in multiple buffer queues for framing to obtain target video frames.

[0230] In some possible implementations, the framing module 502 is used to obtain video frames with the same first timestamp in multiple buffer queues for framing. When obtaining the target video frame, it is specifically used to obtain the video frame at the head position of each buffer queue as the head video frame; obtain the first timestamp of each head video frame; determine whether the first timestamps are the same, and if they are the same, framing the head video frame to obtain the target video frame.

[0231] In some possible implementations, the framing module 502 is specifically used for: if the video data is not multi-channel video data, determining a first duration of the video frame according to the video track information; when the first duration is less than a first preset duration threshold, determining that each target video frame contains a first number of video frames according to the first duration and the first preset duration threshold; determining a first timestamp for each video frame; and selecting the first number of video frames in sequence according to the order corresponding to the first timestamps for framing to obtain a target video frame.

[0232] In some possible implementations, at least one channel of video data includes: one channel of basic layer video data and one channel of enhanced layer video data. Before each channel of video data is stored in a corresponding buffer queue, it is specifically used to: add a target identifier in the enhanced layer video frame; then the decoding module 503 is specifically used to, in the target video frame, identify the enhanced layer video frame and the basic layer video frame according to the target identifier; first decode the basic layer video frame, and then decode the enhanced layer video frame; wherein the basic layer video data includes: multiple basic layer video frames; the enhanced layer video data includes: multiple enhanced layer video frames.

[0233] In some possible implementations, the video track information includes: video metadata; the framing module is specifically used to: framing the video metadata and each video frame to obtain multiple target video frames.

[0234] In some possible implementations, the framing module 502 is specifically used to: framing video frames of the video data to obtain intermediate video frames; framing the video metadata and each intermediate video frame to obtain multiple target video frames.

[0235] In some possible implementations, the multimedia data includes: at least one audio data channel, the track information includes: audio track information, and the audio data includes multiple audio frames; the framing module 502 is specifically used to: determine the second duration of the audio frame according to the audio track information; if the second duration is less than the second preset duration threshold, for each audio data channel, frame the audio frame to obtain multiple target audio frames.

[0236] In some possible implementations, when the framing module 502 is used to frame the audio frames for each channel of audio data to obtain multiple target audio frames, it is specifically used to: determine, based on the second duration and the second preset duration threshold, that each target audio frame contains a second number of audio frames; determine a second timestamp of the audio frame; and select the second number of audio frames in sequence according to the order corresponding to the second timestamps for framing to obtain the target audio frame.

[0237] It should be noted that the device provided in this embodiment can be used to execute the above-mentioned video processing method, and its implementation method and technical effect are similar, which will not be described in detail in this embodiment.

[0238] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0239] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more ASICs (Application Specific Integrated Circuit), or one or more DSPs (Digital Signal Processors), or one or more FPGAs (Field Programmable Gate Arrays), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a CPU or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a SOC (System-on-a-Chip).

[0240] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0241] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the video processing method of any of the above method embodiments is implemented.

[0242] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can obtain the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the video processing method of any of the above method embodiments.

[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0244] For ease of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: The display device comprises: a player and a display screen; The player is configured to: Acquire a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file and multimedia data, wherein the multimedia data includes audio data and video data, and the track information includes video track information; According to the track information, the multimedia data is framed to obtain target multimedia data; Decoding the target multimedia data; Transmitting the decoded target multimedia data to the display screen for display; Determining whether the video data is multi-channel video data according to the video track information; When the multimedia data includes multiple channels of video data, the frame grouping processing method is: storing each channel of video data in a corresponding buffer queue, and grouping the video frames with the same timestamp in each buffer queue; When the multimedia data is one channel of video data, and all video frames are key frames, the frame grouping processing method is: grouping video frames in one channel of video data whose duration is less than a first preset threshold.

2. The display device according to claim 1, characterized in that The multimedia data includes at least one channel of video data, and the video data includes a plurality of video frames; When the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, the player is specifically used to: If the video data is multi-channel video data, creating a buffer queue corresponding to each channel of video data in the multi-channel video data; Storing the video data of each channel in the buffer queue accordingly; In the plurality of buffer queues, video frames with the same first timestamp are obtained for framing to obtain the target video frame.

3. The display device according to claim 2, characterized in that When the player obtains the video frames with the same timestamp from the plurality of buffer queues and groups them to obtain the target video frame, the method is specifically used to: Acquire the video frame at the head position of each buffer queue as the head video frame; Obtaining the first timestamp of each of the team head video frames; Determine whether the first timestamps are the same. If they are the same, group the head video frames to obtain the target video frames.

4. The display device according to claim 2, characterized in that The player is used to: If the video data is not multi-channel video data, determining a first duration of the video frame according to the video track information; When the first duration is less than a first preset duration threshold, determining, according to the first duration and the first preset duration threshold, that each of the target video frames includes the first number of video frames; Determining the first timestamp of each of the video frames; According to the order corresponding to the first timestamps, the video frames of the first number are selected in sequence for framing to obtain the target video frame.

5. The display device according to claim 2, characterized in that The at least one channel of video data includes: a channel of basic layer video data and a channel of enhanced layer video data; before the player stores the respective channels of video data in the buffer queue, the player is specifically configured to: Adding target identification in the enhanced layer video frame; Then, when the player decodes the target multimedia data, it is specifically used to: In the target video frame, identifying the enhancement layer video frame and the base layer video frame according to the target identifier; The base layer video frame is first decoded, and then the enhancement layer video frame is decoded; wherein the base layer video data includes: a plurality of the base layer video frames; the enhancement layer video data includes: a plurality of the enhancement layer video frames.

6. The display device according to claim 2, characterized in that The video track information includes: video metadata; when the player performs frame processing on the multimedia data according to the track information to obtain target multimedia data, it is specifically used to: The video metadata and each of the video frames are grouped to obtain a plurality of the target video frames.

7. The display device according to claim 6, characterized in that When the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, the player is specifically used to: Grouping the video frames of the video data to obtain intermediate video frames; The video metadata is combined with each of the intermediate video frames to obtain a plurality of the target video frames.

8. The display device according to any one of claims 1 to 7, characterized in that: The multimedia data includes: at least one channel of audio data, the track information includes: audio track information, and the audio data includes a plurality of audio frames; When the player performs frame processing on the multimedia data according to the track information to obtain the target multimedia data, the player is specifically used to: Determining a second duration of the audio frame according to the audio track information; If the second duration is less than a second preset duration threshold, the audio frames are grouped for each channel of the audio data to obtain a plurality of target audio frames.

9. The display device according to claim 8, characterized in that The player is specifically used to group the audio frames for each channel of the audio data to obtain multiple target audio frames: Determine, according to the second duration and the second preset duration threshold, that each of the target audio frames includes a second number of audio frames; determining a second timestamp of the audio frame; According to the order corresponding to the second timestamps, the audio frames of the second number are selected in sequence for framing to obtain the target audio frame.

10. A video processing method, characterized in that: Applied to a display device, the video processing method comprises: Acquire a multimedia file, and demultiplex the multimedia file to obtain track information of the multimedia file and multimedia data, wherein the multimedia data includes audio data and video data, and the track information includes video track information; Performing frame processing on the multimedia data according to the track information to obtain target multimedia data; Decoding the target multimedia data; Transmitting the decoded target multimedia data to the display screen for display; Determining whether the video data is multi-channel video data according to the video track information; When the multimedia data includes multiple channels of video data, the frame grouping processing method is: storing each channel of video data in a corresponding buffer queue, and grouping the video frames with the same timestamp in each buffer queue; When the multimedia data is one channel of video data, and all video frames are key frames, the frame grouping processing method is: grouping video frames in one channel of video data whose duration is less than a first preset threshold.

11. The video processing method according to claim 10, characterized in that: The multimedia data includes at least one channel of video data, and the video data includes a plurality of video frames; The multimedia data is framed according to the track information to obtain target multimedia data, including: If the video data is multi-channel video data, creating a buffer queue corresponding to each channel of video data in the multi-channel video data; Storing the video data of each channel in the buffer queue accordingly; In the plurality of buffer queues, video frames with the same first timestamp are obtained for framing to obtain the target video frame.

Images