Audio and video playing method and device and terminal equipment

Abstract

The invention is suitable for the technical field of multimedia, and provides an audio and video playing method and device and terminal device.The method is applied to electronic equipment, an audio and video playing application program configured in the electronic equipment is developed based on a QML cross-platform user interface framework, and the method requests a server for a to-be-played data stream; performing demultiplexing processing on the to-be-played data stream to obtain an original audio data stream and an original video data stream; performing decoding processing on the original audio data stream to obtain target audio data, and performing image processing on the original video data stream to obtain target image data; and in the process of playing the target audio data, synchronously displaying the target image data through a built-in Image control. According to the method, the processing performance of the real-time data stream is enhanced by utilizing the advantage of high loading speed of the Image control, and the low-delay real-time audio and video transmission requirement is met; the audio and video playing application program is developed based on the QML cross-platform user interface framework, so that the cross-platform compatibility of the audio and video playing method is enhanced.

Description

Technical Field

[0001] The present application belongs to the field of multimedia technology, and in particular relates to an audio and video playback method, apparatus, and terminal device. Background Art

[0002] With the rapid development of audio and video coding technologies (such as H.265, AV1) and wireless communication technologies (such as Wi-Fi6, 5G), users' demand for high-resolution, low-latency real-time audio and video transmission has increased significantly.

[0003] However, traditional audio and video playback methods are limited by local file playback and fixed network environments, and have insufficient processing performance for real-time video streams. They are difficult to meet the real-time audio and video transmission needs in mobile scenarios, and adaptation layers need to be developed separately for different platforms. Summary of the Invention

[0004] The embodiments of the present application provide an audio and video playback method, apparatus, and terminal device, which can enhance the processing performance of real-time data streams, meet the requirements of low-latency real-time audio and video transmission, and have strong cross-platform compatibility.

[0005] In a first aspect, an embodiment of the present application provides an audio and video playback method, which is applied to an electronic device, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on a QML cross-platform user interface framework, the method comprising:

[0006] In response to a user-triggered audio or video playback request, requesting a server for a to-be-played data stream corresponding to the audio or video playback request;

[0007] Demultiplexing the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played;

[0008] Decoding the original audio data stream to obtain target audio data, and performing image processing on the original video data stream to obtain target image data, wherein the image processing at least includes decoding processing and image rendering processing;

[0009] During the playback of the target audio data, the target image data is synchronously displayed through the built-in Image control of the audio and video playback application.

[0010] In a possible implementation of the first aspect, performing image processing on an original video data stream to obtain target image data includes:

[0011] The original video data stream is decoded using the FFmpeg library built into the electronic device to obtain the original image data. The FFmpeg library integrates various open source program codes for encoding and decoding.

[0012] Perform image rendering processing on the original image data to obtain target image data.

[0013] In a possible implementation of the first aspect, performing image rendering processing on original image data to obtain target image data includes:

[0014] The OpenGL interface is used to perform image rendering processing on the original image data to obtain the target image data.

[0015] In a possible implementation of the first aspect, synchronously displaying target image data through an Image control built into an audio and video playback application includes:

[0016] Use QML language to convert the target image data into a language to obtain the converted image data;

[0017] The converted image data is synchronously displayed through the Image control.

[0018] In a possible implementation of the first aspect, after requesting the server for the to-be-played data stream corresponding to the audio and video playback request, the method further includes:

[0019] The data stream to be played is stored in a preset buffer data space. The preset buffer data space is used to retrieve the data stream to be played from the preset buffer data space when the target image data is lost, so as to display the target image data according to the retrieved data stream to be played;

[0020] Demultiplexing the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played, including:

[0021] When the size of the stored data in the preset buffer data space reaches the preset storage size, the recvmmsg interface or the I / O interface is used to extract the data stream to be played from the preset buffer data space;

[0022] The data stream to be played is demultiplexed to obtain the original audio data stream and the original video data stream in the data stream to be played.

[0023] In a second aspect, an embodiment of the present application provides an audio and video playback method, applied to a server, the method comprising:

[0024] In response to an audio or video playback request from an electronic device, determining an original audio data stream and an original video data stream corresponding to the audio or video playback request;

[0025] Performing data processing on the original audio data stream and the original video data stream to obtain a data stream to be played, wherein the data processing at least includes key frame interval processing and multiplexing processing;

[0026] The data stream to be played is sent to the electronic device via the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, which is developed based on the QML cross-platform user interface framework. The audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the Image control built into the audio and video playback application during the process of playing the target audio data.

[0027] In a possible implementation of the second aspect, processing the original audio data stream and the original video data stream to obtain the to-be-played data stream includes:

[0028] Performing noise reduction processing on the original audio data stream and the original video data stream respectively to obtain a first audio data stream and a first video data stream;

[0029] Using the server's built-in FFmpeg library to encode the first audio data stream and the first video data stream, respectively, to obtain a second audio data stream and a second video data stream, wherein the FFmpeg library integrates various open source program codes for encoding and decoding;

[0030] Adjusting the key frame interval of the second video data stream according to the network bandwidth fluctuation information of the server to obtain a third video data stream;

[0031] The second audio data stream and the third video data stream are multiplexed to obtain a data stream to be played.

[0032] In a third aspect, an embodiment of the present application provides an audio and video playback device configured in an electronic device, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on a QML cross-platform user interface framework, and the device includes:

[0033] A request module, configured to respond to a user-triggered audio or video playback request and request a server for a to-be-played data stream corresponding to the audio or video playback request;

[0034] A first processing module is configured to perform demultiplexing on the data stream to be played to obtain an original audio data stream and an original video data stream in the data stream to be played;

[0035] a second processing module, configured to decode the original audio data stream to obtain target audio data, and perform image processing on the original video data stream to obtain target image data, wherein the image processing includes at least decoding processing and image rendering processing;

[0036] The playing module is used to synchronously display the target image data through the built-in Image control of the audio and video playing application during the process of playing the target audio data.

[0037] In a fourth aspect, an embodiment of the present application provides an audio and video playback device, which is configured on a server, and the device includes:

[0038] a determination module, configured to determine, in response to an audio or video playback request of an electronic device, an original audio data stream and an original video data stream corresponding to the audio or video playback request;

[0039] A third processing module is configured to perform data processing on the original audio data stream and the original video data stream to obtain a data stream to be played, wherein the data processing includes at least key frame interval processing and multiplexing processing;

[0040] A sending module is used to send the data stream to be played to the electronic device through the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on the QML cross-platform user interface framework. The audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the Image control built into the audio and video playback application during the process of playing the target audio data.

[0041] In a fifth aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of any one of the first aspect or the second aspect is implemented.

[0042] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the first aspect or the second aspect.

[0043] In a seventh aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the methods of the first or second aspects above.

[0044] The embodiment of the present application provides an audio and video playback method, device and terminal device, the method is applied to an electronic device, the electronic device is configured with an audio and video playback application, the audio and video playback application is developed based on a cross-platform user interface framework of QML, the method includes: in response to an audio and video playback request triggered by a user, requesting a data stream to be played corresponding to the audio and video playback request from a server; demultiplexing the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played; decoding the original audio data stream to obtain target audio data, and performing image processing on the original video data stream to obtain target image data, the image processing at least includes decoding processing and image rendering processing; in the process of playing the target audio data, the target image data is synchronously displayed through the Image control built into the audio and video playback application. Using the above technical solution, in the process of playing the target audio data, the target image data is synchronously displayed through the Image control built into the audio and video playback application, which can enhance the processing performance of the real-time data stream by taking advantage of the fast image loading speed of the Image control and meet the low-latency real-time audio and video transmission requirements; at the same time, because the audio and video playback application is developed based on a cross-platform user interface framework of QML, the cross-platform compatibility of the audio and video playback method is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of an audio and video playback method provided by an embodiment of the present application;

[0047] Figure 2 This is a flowchart of an audio and video playback method provided by another embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of the overall architecture of an audio and video playback method provided by an embodiment of the present application;

[0049] Figure 4 This is a structural block diagram of an audio and video playback device provided by an embodiment of the present application;

[0050] Figure 5 This is a structural block diagram of an audio and video playback device provided by another embodiment of the present application;

[0051] Figure 6This is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0053] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0054] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0055] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0056] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0057] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0058] It can be said that the development efficiency of traditional audio and video playback methods is low, and the current official MediaPlayer and VideoOutput have insufficient processing performance for real-time video streams; current wireless audio and video software lacks native support for real-time streaming protocols (such as RTSP and WebRTC); cross-platform compatibility is poor, and separate adaptation layers need to be developed for different platforms.

[0059] Based on this, the embodiment of the present application provides an audio and video playback method, which, through cross-platform capabilities and multimedia processing technology, can systematically solve core problems such as performance bottlenecks, lack of real-time performance, and poor cross-platform compatibility in traditional audio and video development.

[0060] Figure 1 This is a flow chart of an audio and video playback method provided by an embodiment of the present application. As an example and not a limitation, the method can be applied to an electronic device, in which an audio and video playback application is configured. The audio and video playback application is developed based on the QML cross-platform user interface framework (i.e., QtQuick framework), such as Figure 1 As shown, the method includes:

[0061] S101 : In response to an audio or video playback request triggered by a user, request a server for a to-be-played data stream corresponding to the audio or video playback request.

[0062] A pre-established communication connection between the electronic device and the server can be used. For example, an asynchronous I / O model can be constructed using epoll+non-blocking sockets to implement TCP fast retransmit or UDP forward error correction. epoll+non-blocking sockets is a widely used network programming model that sets the socket to non-blocking mode, making it particularly suitable for high-concurrency scenarios.

[0063] When a user wants to play audio and video data, he can generate a corresponding audio and video playback request by performing a specific trigger action on the electronic device. For example, he can trigger the generation of an audio and video playback request by clicking a play button in the audio and video playback application of the electronic device. The audio and video playback application can then respond to this audio and video playback request and request the server for the data stream to be played corresponding to the audio and video playback request. The data stream to be played may refer to the original data stream returned by the server to the electronic device based on the audio and video playback request.

[0064] S102: Demultiplex the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played.

[0065] S103: Decode the original audio data stream to obtain target audio data, and perform image processing on the original video data stream to obtain target image data. The image processing at least includes decoding processing and image rendering processing.

[0066] After receiving the data stream to be played returned by the server, the audio and video playback application can perform a series of data processing on the data stream to be played to obtain the target audio data and target image data for final playback. For example, demultiplexing processing can be the process of separating a multimedia container file containing audio, video or other data streams into individual independent streams. These streams can be video streams, audio streams, subtitle streams, etc., so that the audio data stream and video data stream can be processed separately later. Therefore, this embodiment can use the FFmpeg library to demultiplex the data stream to be played, separate the original audio data stream and the original video data stream in the data stream to be played, and then process the obtained original audio data stream and original video data stream respectively to obtain the corresponding target audio data and target image data.

[0067] The specific means of obtaining the target audio data and target image data in this embodiment are not limited. For example, different neural network models can be used to input the original audio data stream and the original video data stream into their respective neural network models to obtain the corresponding target audio data and target image data. Alternatively, the target audio data and target image data can be obtained separately by other means, such as using hard decoding to decode the original audio data stream to obtain the target audio data to improve performance.

[0068] As a feasible implementation method, image processing is performed on the original video data stream to obtain target image data, including: using the FFmpeg library built into the electronic device to decode the original video data stream to obtain original image data, wherein the FFmpeg library integrates various open source program codes for encoding and decoding; and performing image rendering processing on the original image data to obtain target image data.

[0069] Among them, the FFmpeg (Fast Forward moving picture experts group) library integrates various open source program codes for encoding and decoding. The open source program codes are also program files that can realize processing operations such as encoding, decoding, multiplexing, demultiplexing, streaming, and filtering of image information. In addition, the FFmpeg library supports cross-platform and can be used on Windows, MacOS, Linux, Android, and iOS.

[0070] Furthermore, for image color space conversion, the decoded raw image data can be rendered. This operation can be accelerated using the OpenCL interface. For example, the OpenGL interface can be used to render the raw image data to obtain the target image data. This can increase the speed of image processing and enhance the performance of audio and video playback.

[0071] Alternatively, you can customize TextureProvider to inherit from QSGTextureProvider. Using a cross-thread signal update mechanism, you can skip frame updates when the QML engine is rendering, thereby improving performance. Alternatively, you can directly manipulate OpenGL textures through the Scene Graph of a cross-platform application development framework for further optimized display and sub-millisecond rendering latency, making it particularly suitable for scenarios with high real-time requirements, such as drone image transmission and industrial monitoring. The Scene Graph can be the underlying rendering architecture of a cross-platform application development framework, enabling high-performance rendering.

[0072] As a feasible implementation method, this embodiment can further improve performance by selectively developing a multi-level cache mechanism to skip repeated processing of static background areas.

[0073] S104: During the process of playing the target audio data, the target image data is synchronously displayed through the built-in Image control of the audio and video playback application.

[0074] Specifically, in the field of displaying images, the Image control has significant advantages over MediaPlayer and VideoOutput in terms of functional positioning, resource usage, and implementation logic. For example, the Image control has lower resource consumption and better performance. On the one hand, the Image control is lightweight and has a low memory usage. That is, the Image control only needs to load a single frame of image data and does not need to process complex logic such as continuous frame decoding and audio and video synchronization. The memory usage is usually in the KB level. In contrast, MediaPlayer needs to continuously decode frame data when playing videos, and the memory usage can reach tens of MB or even higher, and it needs to occupy CPU and GPU resources for real-time rendering. On the other hand, the Image control has a fast startup speed and high rendering efficiency. That is, when loading images, the Image control does not need to parse complex media container formats (such as MP4 video tracks and audio tracks). It directly reads the image pixel data and renders it, and the response time is usually in milliseconds. For example, when sliding through pictures in the album app, the Image control can quickly load the next picture, while when MediaPlayer is used for single-frame display, it will cause lag due to format parsing delays.

[0075] Optionally, an independent decoding thread can be created at the decoding level, and the FFmpeg library can be used for hardware decoding. After the video is decoded, the texture handle can be directly passed to the QML engine of the cross-platform application development framework for use on different platforms.

[0076] At the same time, the Image control's interface is concise and easier to use. On the one hand, its API design is more tailored to the needs of static images. Specifically, its core interface is designed around "image loading, displaying, and interacting," with methods like setImageResource() (loading resources), setScaleType() (setting the scale mode), and setOnClickListener() (click events), resulting in a low learning curve. This compares to the MediaPlayer interface, which includes dynamic control methods like prepare() (prepare for playback), start() (start), and seekTo() (jump progress), resulting in redundant interfaces for static images. Furthermore, the Image control offers greater cross-platform compatibility. Most platforms (such as Android, iOS, and the Web) offer more basic and unified support for the Image control, while the MediaPlayer interface and functionality vary significantly across platforms (e.g., Android's MediaPlayer vs. iOS's AVPlayer), requiring additional adaptation.

[0077] Therefore, in this step, during the playback of the target audio data, the rendered target image data can be exposed to the Image control built into the audio and video playback application through the QQuickImageProvider class, so that the target image data can be synchronously displayed through the Image control instead of outputting the image through MediaPlayer and VideoOutput. On this basis, the processing performance of the real-time data stream can be enhanced to meet the low-latency real-time audio and video transmission requirements.

[0078] As a feasible implementation method, the target image data is synchronously displayed through the Image control built into the audio and video playback application, including: using QML language to convert the target image data to obtain converted image data; and synchronously displaying the converted image data through the Image control.

[0079] Among them, QML language (Qt Meta-Object Language or Qt Modeling Language) can be a declarative UI programming language in a cross-platform application development framework, used to describe the user interface of a program. It should be noted that before the image rendering process of this embodiment, the C++ programming language can be used for description, and after the image rendering process of this embodiment, the QML language can be used. Therefore, this embodiment can communicate through the QQuickImageProvider class, use the QML language to convert the target image data into a language, and obtain the converted image data; and the converted image data is synchronously displayed through the Image control.

[0080] This embodiment provides an audio and video playback method that, during the process of playing target audio data, synchronously displays target image data through the Image control built into the audio and video playback application. This method can utilize the advantage of the Image control's fast image loading speed to enhance the processing performance of real-time data streams and meet the requirements of low-latency real-time audio and video transmission. At the same time, since the audio and video playback application is developed based on the QML cross-platform user interface framework, the cross-platform compatibility of the audio and video playback method is enhanced.

[0081] As a feasible implementation method, after requesting the server for the data stream to be played corresponding to the audio and video playback request, the method also includes: storing the data stream to be played in a preset buffer data space, and the preset buffer data space is used to re-acquire the data stream to be played from the preset buffer data space when the target image data is lost, so as to display the target image data according to the re-acquired data stream to be played.

[0082] In a specific implementation, the receiving method can be optimized. For example, after receiving the data stream to be played returned by the server, the data stream to be played can be preferentially stored in a preset buffer data space, so that in the subsequent data stream transmission process, when data is temporarily lost, the data in the preset buffer data space can be relied on to continue playing. In this embodiment, the preset buffer data space can be a ring buffer based on QtSharedMemory, which contains 4 128MB blocks and uses atomic operations to achieve lock-free reading and writing. For example, the capacity of the ring buffer needs to meet 2 seconds of video data. On this basis, the use of a ring buffer can handle network jitter and optimize the occurrence of video freezes.

[0083] At the same time, this embodiment can extract all the data in the preset buffer data space when the preset buffer data space reaches a specific situation, so as to perform subsequent data stream processing. Exemplarily, the data stream to be played is demultiplexed to obtain the original audio data stream and the original video data stream in the data stream to be played, including: when the stored data size of the preset buffer data space reaches the preset storage size, the recvmmsg interface or the I / O interface is used to extract the data stream to be played from the preset buffer data space; the data stream to be played is demultiplexed to obtain the original audio data stream and the original video data stream in the data stream to be played. Among them, the preset storage size can be a pre-set storage size, such as the preset storage size can be the maximum storage space size of the preset buffer data space, or it can be less than the maximum storage space size, etc.

[0084] Zero-copy technology can be used to improve performance when extracting the data stream to be played. Zero-copy technology reduces the number of data copies, primarily to reduce CPU and memory usage. For example, in traditional data transmission, data typically needs to be copied from a source memory area to a target memory area. This process requires CPU participation and consumes a large amount of memory bandwidth. Zero-copy technology improves data transmission efficiency by reducing or eliminating these copy operations.

[0085] Specifically, different methods can be used to extract the data stream to be played based on the specific operating system of the electronic device. For example, for Linux operating systems, the recvmmsg interface can be used to batch extract the data stream to be played. For Windows operating systems, the I / O interface can be used to extract the data stream to be played from a preset buffered data space. This can improve data transmission efficiency.

[0086] Figure 2 This is a flowchart of an audio and video playback method provided by another embodiment of the present application. As an example and not a limitation, this method can be applied to a server, such as Figure 2 As shown, the method includes:

[0087] S201: In response to an audio or video playback request from an electronic device, determine an original audio data stream and an original video data stream corresponding to the audio or video playback request.

[0088] S202: Process the original audio data stream and the original video data stream to obtain a data stream to be played. The data processing at least includes key frame interval processing and multiplexing processing.

[0089] After the server obtains the original audio data stream and original video data stream to be transmitted, certain data processing can be performed on the original audio data stream and the original video data stream to realize the protocol encapsulation of the data stream. The specific data processing process may at least include key frame interval processing and multiplexing processing. On this basis, other types of data processing operations can also be performed according to actual needs.

[0090] As a feasible implementation method, the original audio data stream and the original video data stream are processed to obtain the data stream to be played, including: performing noise reduction processing on the original audio data stream and the original video data stream respectively to obtain the first audio data stream and the first video data stream; using the FFmpeg library built into the server to encode the first audio data stream and the first video data stream respectively to obtain the second audio data stream and the second video data stream, wherein the FFmpeg library integrates various open source program codes for encoding and decoding; adjusting the key frame interval of the second video data stream according to the network bandwidth fluctuation information of the server to obtain the third video data stream; and multiplexing the second audio data stream and the third video data stream to obtain the data stream to be played.

[0091] Among them, before the encoding process, the original audio data stream and the original video data stream can be subjected to noise reduction processing respectively. For example, the image noise can be eliminated by using the non-local means filter (NLMeans) filter; then the server side can use the FFmpeg library to perform encoding processing in different formats on the first audio data stream and the first video data stream respectively. The encoding processing supports common formats such as H264 and H265. Furthermore, after the second audio data stream and the second video data stream are obtained through encoding processing, the key frame interval can be dynamically adjusted, such as automatically adjusting the key frame interval according to fluctuations in network bandwidth.

[0092] Furthermore, this embodiment can balance image quality and bandwidth by setting two parameters before encoding the video stream: variable bitrate (VBR) and constrained variable bitrate (CVBR). This allows the encoder to prioritize VBR for image quality and dynamically allocate bitrates. When the bitrate exceeds CVBR, the compiler will constrain the bitrate to be lower than the preset peak bitrate. This allows for controlled image quality and an improved user experience for audio and video playback.

[0093] Furthermore, if the server supports H.264 / H.265 8K encoding, this embodiment can also accelerate the encoding process by enabling -hwaccelcuda, thereby improving the processing speed.

[0094] S203. Send the data stream to be played to the electronic device via the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, the audio and video playback application is developed based on the QML cross-platform user interface framework, and the audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the Image control built into the audio and video playback application during the process of playing the target audio data.

[0095] Optionally, this embodiment can use the UDP communication protocol as the main transport layer to encapsulate the RTP / RTSP protocol, and in the event of frame loss, it can cooperate with the QUIC protocol to implement a fast retransmission mechanism to balance real-time performance and reliability.

[0096] Figure 3 This is a schematic diagram of the overall architecture of an audio and video playback method provided by an embodiment of the present application, referring to Figure 3 The streaming media server (i.e., the server) can first call the audio and video data to be transmitted (i.e., determine the original audio data stream and original video data stream corresponding to the audio and video playback request), encode the audio and video data / encapsulate the protocol. For example, the server can use the FFmpeg library to encode the audio and video data separately (supporting common formats such as H264 and H265), and then send the processed audio and video streams to the client through a TCP / UDP connection.

[0097] This client can be an audio and video playback application configured in an electronic device. The audio and video playback application can be developed based on the QML cross-platform user interface framework, and the electronic device has a built-in FFmpeg library. After the client receives the processed audio and video stream sent by the server, it can use the FFmpeg library to demultiplex the audio and video streams, use an audio and video decoder to decode the demultiplexed data stream, and then perform video frame post-processing on the decoded image. For example, OpenCL can be used for accelerated rendering to improve performance; finally, the rendered image can be registered as a frame image that can be called and displayed by QML, so that the rendered image can be exposed to the QML Image control through the QQuickImageProvider class for image display.

[0098] From the above description, it can be found that the audio and video playback method of this embodiment provides a real-time audio and video playback and processing solution based on the QtQuick framework, which is suitable for embedded software usage scenarios with high real-time requirements such as drone image transmission and industrial monitoring. The specific solution is to push audio and video streams through the server's deep integration of wireless transmission protocols such as TCP / UDP. The client receives and parses the audio and video streams frame by frame through the FFmpeg library, and references each frame of the parsed video image to the QML Image control through shared memory or reference counting with the QQuickImageProvider type, and uses the Image control instead of MediaPlayer and VideoOutput for video output. Among them, because it is compiled with the QtQuick framework, it has strong cross-platform compatibility and can be flexibly compatible with various platforms developed based on Linux and Windows.

[0099] Corresponding to the audio and video playback method of the above embodiment, Figure 4 This is a structural block diagram of an audio and video playback device provided in one embodiment of the present application, which is configured in an electronic device. The electronic device is equipped with an audio and video playback application, which is developed based on the QML cross-platform user interface framework. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0100] Reference Figure 4 , the device comprises:

[0101] The request module 301 is used to respond to the audio and video playback request triggered by the user and request the server for the to-be-played data stream corresponding to the audio and video playback request;

[0102] The first processing module 302 is used to demultiplex the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played;

[0103] The second processing module 303 is configured to decode the original audio data stream to obtain target audio data, and perform image processing on the original video data stream to obtain target image data, where the image processing includes at least decoding processing and image rendering processing;

[0104] The playing module 304 is configured to synchronously display the target image data through the built-in Image control of the audio and video playing application during the process of playing the target audio data.

[0105] This embodiment provides an audio and video playback device, which, in response to a user-triggered audio and video playback request, requests a server for a data stream to be played corresponding to the audio and video playback request via a request module; demultiplexes the data stream to be played via a first processing module to obtain an original audio data stream and an original video data stream in the data stream to be played; decodes the original audio data stream via a second processing module to obtain target audio data, and performs image processing on the original video data stream to obtain target image data, wherein the image processing includes at least decoding and image rendering; and synchronously displays the target image data via an Image control built into the audio and video playback application during playback of the target audio data via the playback module. Using this device, during playback of the target audio data, the target image data is synchronously displayed via an Image control built into the audio and video playback application, thereby enhancing the processing performance of real-time data streams by utilizing the advantage of the Image control's fast image loading speed, thereby meeting the requirements of low-latency real-time audio and video transmission. Furthermore, because the audio and video playback application is developed based on a QML cross-platform user interface framework, the cross-platform compatibility of the audio and video playback method is enhanced.

[0106] Optionally, the second processing module includes:

[0107] A decoding processing unit, configured to decode the original video data stream using the FFmpeg library built into the electronic device to obtain original image data, wherein the FFmpeg library integrates various open source program codes for encoding and decoding;

[0108] The rendering processing unit is used to perform image rendering processing on the original image data to obtain target image data.

[0109] Optionally, the rendering processing unit is specifically configured to:

[0110] The OpenGL interface is used to perform image rendering processing on the original image data to obtain the target image data.

[0111] Optionally, the playback module is specifically configured to:

[0112] Use QML language to convert the target image data into a language to obtain the converted image data;

[0113] The converted image data is synchronously displayed through the Image control.

[0114] Optionally, the audio and video playback device provided in this embodiment further includes:

[0115] a storage module configured to, after requesting a to-be-played data stream corresponding to an audio or video playback request from a server, store the to-be-played data stream in a preset buffer data space, wherein the preset buffer data space is used to retrieve the to-be-played data stream from the preset buffer data space when target image data is lost, so as to display the target image data based on the retrieved to-be-played data stream;

[0116] The first processing module is specifically configured to:

[0117] When the size of the stored data in the preset buffer data space reaches the preset storage size, the recvmmsg interface or the I / O interface is used to extract the data stream to be played from the preset buffer data space;

[0118] The data stream to be played is demultiplexed to obtain the original audio data stream and the original video data stream in the data stream to be played.

[0119] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0120] Corresponding to the audio and video playback method of the above embodiment, Figure 5 This is a structural block diagram of an audio and video playback device provided in another embodiment of the present application, which is configured on a server. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0121] Reference Figure 5 , the device comprises:

[0122] The determining module 401 is configured to determine, in response to an audio or video playback request of an electronic device, an original audio data stream and an original video data stream corresponding to the audio or video playback request;

[0123] The third processing module 402 is used to process the original audio data stream and the original video data stream to obtain a data stream to be played, wherein the data processing includes at least key frame interval processing and multiplexing processing;

[0124] The sending module 403 is used to send the data stream to be played to the electronic device via the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on the QML cross-platform user interface framework. The audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the Image control built into the audio and video playback application during the process of playing the target audio data.

[0125] The present embodiment provides an audio and video playback device, which, in response to an audio and video playback request of an electronic device, determines an original audio data stream and an original video data stream corresponding to the audio and video playback request through a determination module; processes the original audio data stream and the original video data stream through a third processing module to obtain a data stream to be played, wherein the data processing includes at least key frame interval processing and multiplexing processing; and sends the data stream to be played to the electronic device through a sending module via a TCP communication protocol or a UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, the audio and video playback application is developed based on a QML cross-platform user interface framework, and the audio and video playback application is used to process the data stream to be played to obtain target audio data and target image data, so that during the process of playing the target audio data, the target image data is synchronously displayed through an Image control built into the audio and video playback application. By using this device, the target image data is synchronously displayed by the electronic device side according to the Image control built into the audio and video playback application, and the processing performance of the real-time data stream can be enhanced by taking advantage of the fast image loading speed of the Image control, thereby meeting the low-latency real-time audio and video transmission requirements; at the same time, because the audio and video playback application is developed based on the QML cross-platform user interface framework, the cross-platform compatibility of the audio and video playback method is enhanced.

[0126] Optionally, the third processing module is specifically configured to:

[0127] Performing noise reduction processing on the original audio data stream and the original video data stream respectively to obtain a first audio data stream and a first video data stream;

[0128] Using the server's built-in FFmpeg library to encode the first audio data stream and the first video data stream, respectively, to obtain a second audio data stream and a second video data stream, wherein the FFmpeg library integrates various open source program codes for encoding and decoding;

[0129] Adjusting the key frame interval of the second video data stream according to the network bandwidth fluctuation information of the server to obtain a third video data stream;

[0130] The second audio data stream and the third video data stream are multiplexed to obtain a data stream to be played.

[0131] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0133] Figure 6 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present application. Figure 6 As shown, the terminal device 500 of this embodiment includes: at least one processor 502 ( Figure 6 Only one is shown in the figure) a processor, a memory 501, and a computer program 503 stored in the memory 501 and executable on at least one processor 502. When the processor 502 executes the computer program 503, the steps in the control method embodiment of any of the above-mentioned applications are implemented.

[0134] The terminal device 500 may be a computing device such as a desktop computer, a notebook, a palmtop computer, or a server. The terminal device may include, but is not limited to, a processor 502 and a memory 501. Those skilled in the art will appreciate that Figure 6 This is merely an example of the terminal device 500 and does not constitute a limitation on the terminal device 500 . The terminal device 500 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 500 may also include input and output devices, network access devices, etc.

[0135] The processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0136] In some embodiments, the memory 501 may be an internal storage unit of the terminal device 500, such as a hard disk or memory of the terminal device 500. In other embodiments, the memory 501 may also be an external storage device of the terminal device 500, such as a plug-in hard disk equipped on the terminal device 500, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Furthermore, the memory 501 may also include both an internal storage unit of the terminal device 500 and an external storage device. The memory 501 is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as program code of a computer program. The memory 501 may also be used to temporarily store data that has been output or is about to be output.

[0137] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 502, the steps in the above-mentioned method embodiments can be implemented.

[0138] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 502, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include at least: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable storage medium cannot be an electric carrier signal or a telecommunication signal.

[0140] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0141] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal devices and methods can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0143] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0144] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An audio and video playback method, characterized in that: Applied to an electronic device, the electronic device is equipped with an audio and video playback application, the audio and video playback application is developed based on a QML cross-platform user interface framework, the method includes: In response to a user-triggered audio or video playback request, requesting a server for a to-be-played data stream corresponding to the audio or video playback request; Demultiplexing the data stream to be played to obtain an original audio data stream and an original video data stream in the data stream to be played; Decoding the original audio data stream to obtain target audio data, and performing image processing on the original video data stream to obtain target image data, wherein the image processing includes at least decoding processing and image rendering processing; During the process of playing the target audio data, the target image data is synchronously displayed through the Image control built into the audio and video playback application.

2. The audio and video playback method according to claim 1, wherein: The performing image processing on the original video data stream to obtain target image data includes: Decoding the original video data stream using a built-in FFmpeg library of the electronic device to obtain original image data, wherein the FFmpeg library integrates various open source program codes for encoding and decoding; Perform image rendering processing on the original image data to obtain target image data.

3. The audio and video playback method according to claim 2, wherein: The performing image rendering processing on the original image data to obtain target image data includes: The original image data is subjected to image rendering processing using the OpenGL interface to obtain target image data.

4. The audio and video playback method according to claim 1, wherein: The synchronously displaying the target image data through the built-in Image control of the audio and video playback application comprises: Performing language conversion on the target image data using the QML language to obtain converted image data; The converted image data is synchronously displayed through the Image control.

5. The audio and video playback method according to any one of claims 1 to 4, wherein: After requesting the server for the to-be-played data stream corresponding to the audio and video playback request, the method further includes: storing the data stream to be played in a preset buffer data space, wherein the preset buffer data space is used to retrieve the data stream to be played from the preset buffer data space when the target image data is lost, so as to display the target image data according to the retrieved data stream to be played; The demultiplexing process of the data stream to be played to obtain the original audio data stream and the original video data stream in the data stream to be played includes: When the size of the stored data in the preset buffer data space reaches the preset storage size, extracting the to-be-played data stream from the preset buffer data space using a recvmmsg interface or an I / O interface; Demultiplexing is performed on the data stream to be played to obtain an original audio data stream and an original video data stream in the data stream to be played.

6. An audio and video playback method, characterized in that: Applied to a server, the method includes: In response to an audio or video playback request from an electronic device, determining an original audio data stream and an original video data stream corresponding to the audio or video playback request; Performing data processing on the original audio data stream and the original video data stream to obtain a data stream to be played, wherein the data processing at least includes key frame interval processing and multiplexing processing; The data stream to be played is sent to the electronic device via the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on the QML cross-platform user interface framework. The audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the built-in Image control of the audio and video playback application during the process of playing the target audio data.

7. The audio and video playback method according to claim 6, wherein: The processing of the original audio data stream and the original video data stream to obtain a data stream to be played includes: Performing noise reduction processing on the original audio data stream and the original video data stream respectively to obtain a first audio data stream and a first video data stream; Using a built-in FFmpeg library of the server to encode the first audio data stream and the first video data stream respectively to obtain a second audio data stream and a second video data stream, wherein the FFmpeg library integrates various open source program codes for encoding and decoding; adjusting a key frame interval of the second video data stream according to the network bandwidth fluctuation information of the server to obtain a third video data stream; The second audio data stream and the third video data stream are multiplexed to obtain a data stream to be played.

8. An audio and video playback device, characterized in that: The device is configured in an electronic device, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on a QML cross-platform user interface framework. The device includes: A request module, configured to respond to a user-triggered audio or video playback request and request a server for a to-be-played data stream corresponding to the audio or video playback request; A first processing module is configured to perform demultiplexing on the data stream to be played to obtain an original audio data stream and an original video data stream in the data stream to be played; a second processing module, configured to decode the original audio data stream to obtain target audio data, and perform image processing on the original video data stream to obtain target image data, wherein the image processing includes at least decoding processing and image rendering processing; The playing module is used to synchronously display the target image data through the built-in Image control of the audio and video playing application during the process of playing the target audio data.

9. An audio and video playback device, characterized in that: Configured on a server, the device includes: a determination module, configured to determine, in response to an audio or video playback request of an electronic device, an original audio data stream and an original video data stream corresponding to the audio or video playback request; A third processing module is configured to perform data processing on the original audio data stream and the original video data stream to obtain a data stream to be played, wherein the data processing includes at least key frame interval processing and multiplexing processing; A sending module is used to send the data stream to be played to the electronic device through the TCP communication protocol or the UDP communication protocol, wherein the electronic device is configured with an audio and video playback application, and the audio and video playback application is developed based on the QML cross-platform user interface framework. The audio and video playback application is used to obtain target audio data and target image data based on the data stream to be played, so as to synchronously display the target image data through the Image control built into the audio and video playback application during the process of playing the target audio data.

10. A terminal device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the terminal device implements the method according to any one of claims 1 to 7.

Images