Data processing methods, apparatus, equipment and media

Abstract

This application provides a data processing method, apparatus, device, and medium. The method specifically includes: acquiring target media data; decoding video data in the target media data using hardware decoding technology or a first network intermediate code; wherein the first network intermediate code is intermediate code compiled from a first preset language code possessing preset video encoding characteristics, and the first network intermediate code can run in a browser environment; decoding audio data in the target media data using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code possessing preset audio encoding characteristics, and the second network intermediate code can run in a browser environment. This application embodiment can support more audio and video codec types and encoding characteristics.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] WebRTC (Web Real-Time Communications) is a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between browsers without the need for an intermediary, enabling the transmission of video data, audio streams, or other arbitrary data.

[0003] In applications such as live streaming and cloud rendering, clients can use WebRTC technology to pull media data such as video data and / or audio streams from the server; and use WebRTC technology to decode and render the media data to enable playback.

[0004] In practical applications, WebRTC technology has certain limitations. For example, live streaming scenarios widely use AAC (Advanced Audio Coding) audio encoding and B-frame (bidirectional prediction frame) encoding features, but WebRTC technology does not support AAC audio encoding and B-frame encoding features. Similarly, cloud rendering scenarios typically use HEVC (High Efficiency Video Coding) format, which WebRTC technology does not support. Summary of the Invention

[0005] This application provides a data processing method that can support more audio and video codec types and encoding characteristics.

[0006] Correspondingly, embodiments of this application also provide a data processing device, an electronic device, and a storage medium to implement and apply the above-described methods.

[0007] To address the aforementioned problems, this application discloses a data processing method, the method comprising:

[0008] Acquire target media data;

[0009] The video data in the target media data is decoded using hardware decoding technology or a first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding based on a hardware interface; the first network intermediate code is intermediate code compiled from a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment.

[0010] The audio data in the target media data is decoded using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code with decoding capabilities based on preset audio encoding features, and the second network intermediate code can run in a browser environment.

[0011] To address the aforementioned problems, this application discloses a data processing apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire target media data;

[0013] The video decoding module is used to decode video data in the target media data using hardware decoding technology or a first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding according to a hardware interface; the first network intermediate code is intermediate code compiled from a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment.

[0014] The audio decoding module is used to decode the audio data in the target media data using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code with preset audio encoding characteristics and capable of running in a browser environment.

[0015] Optionally, the acquisition module includes:

[0016] The first acquisition module is used to acquire media data from the server according to the data channel; the data channel is a communication link based on web multimedia communication technology, and the mode of the data channel is unreliable transmission mode.

[0017] The determination module is used to determine the target media data based on the media data obtained from the server.

[0018] Optionally, the first acquisition module includes:

[0019] The communication protocol determination module is used to determine the communication protocol corresponding to the received data;

[0020] The second acquisition module is used to treat the received data corresponding to the first communication protocol as media data when the communication protocol is the first communication protocol.

[0021] Optionally, the first communication protocol is the UDP protocol.

[0022] Optionally, the device further includes:

[0023] A custom data acquisition module is used to treat the received data corresponding to the second communication protocol as custom data when the communication protocol is the second communication protocol.

[0024] Optionally, the audio decoding module includes:

[0025] The initialization module is used to initialize multiple audio decoding threads using the main thread.

[0026] The task distribution module is used to send the audio decoding task corresponding to the audio data in the target media data to the audio decoding thread in the idle state using the main thread, so that the audio decoding thread in the idle state can decode the audio decoding task using the second network intermediate code; wherein, one instance of the second network intermediate code corresponds to one audio decoding task.

[0027] Optionally, the determining module includes:

[0028] The Quality of Service (QoS) processing module is used to process media data obtained from the server using third-party network intermediate code to obtain target media data. The third-party network intermediate code is intermediate code compiled from a third preset language code with QoS processing capabilities, and the third-party network intermediate code can run in a browser environment.

[0029] To address the aforementioned problems, this application discloses an electronic device, including: a processor; and a memory storing executable code thereon, wherein when the executable code is executed, the processor performs the method as described in any of the above embodiments.

[0030] To address the aforementioned issues, embodiments of this application disclose one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform the method as described in any of the above embodiments.

[0031] The embodiments of this application have the following advantages:

[0032] This application embodiment utilizes hardware decoding technology or a first network intermediate code to decode video data in the target media data, and utilizes a second network intermediate code to decode audio data in the target media data. Since hardware decoding technology uses hardware to perform decoding based on a hardware interface, this application embodiment can overcome the limitation of video encoding formats supported by WebRTC technology, for example, it can support the HEVC format video encoding type. Because the first network intermediate code can be intermediate code compiled from a first preset language code with decoding capabilities possessing preset video encoding characteristics, this application embodiment can support preset video encoding characteristics not supported by WebRTC technology, such as the HEVC format. Therefore, this application embodiment can increase the applicability of video encoding types.

[0033] This application embodiment utilizes a second network intermediate code to decode the audio data in the target media data. Since this second network intermediate code is compiled from a second preset language code possessing decoding capabilities with preset audio encoding features, this application embodiment can support preset audio encoding features not supported by WebRTC technology, such as the AAC format. Therefore, this application embodiment can expand the applicability of audio encoding types.

[0034] In summary, the embodiments of this application utilize hardware decoding technology, a first network intermediate code, and a second network intermediate code to take over the audio and video decoding part handled by WebRTC technology, which can expand and support more audio and video codec types and encoding characteristics. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the structure of a data processing system 100 according to an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of a client in related technologies;

[0037] Figure 3 This is a flowchart of the steps of a data processing method according to an embodiment of this application;

[0038] Figure 4 This is a schematic flowchart of a data processing apparatus according to an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of a data processing system according to an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an exemplary device provided in one embodiment of this application. Detailed Implementation

[0042] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] The data processing method of this application embodiment can be used to transmit audio / video data. The data processing method of this application embodiment can be applied to application scenarios such as live streaming, cloud rendering, teleconferencing, video chat, and VoIP. Among these, the live streaming scenario can include: RTS (Real-time Streaming) scenario.

[0044] Cloud rendering places 3D (three-dimensional) programs on a remote server for rendering. Users access resources by clicking a "cloud rendering" button on a client and using a high-speed internet connection. The corresponding cloud rendering command is sent from the client, and the server executes the rendering task accordingly. The rendered result is then sent back to the client for display. The cloud rendering command can carry scene files and auxiliary files. Scene files include, for example, models created by the user using 3D modeling software, the lighting setup, and the materials created. Auxiliary files include, for example, reference files, texture files, proxy files, photon files, RGB values, motion caches, hair caches, light caches, fluid caches, particle caches, and cache files generated by rendering software plugins. The cloud rendering command can also carry rendering parameters, such as the rendering software version used, the filename of the rendered output image, the resolution of the rendered output image, the format of the rendered output image, frame settings (including the start frame, end frame, and the option to render a sequence of frames, detailed frames, or single frames), output image size (width, height), and camera settings. The rendered result transmitted by the server can also be used as audio / video data.

[0045] The audio / video data transmitted in this embodiment can be a type of media stream, which can be understood as an ordered sequence of bytes with a start and an end.

[0046] The embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0047] Reference Figure 1 The diagram illustrates the structure of a data processing system 100 according to an embodiment of this application. Figure 1As shown, the system 100 specifically includes: a playback client 110, a server 120, and a broadcast client 130. The server 120 communicates with both the playback client 110 and the broadcast client 130. The broadcast client 130 sends collected data to the server 120; this process can be a push streaming process, where the broadcast client 130 continuously sends real-time media data to the server 120 during audio / video acquisition. The playback client 110 retrieves media data stored on the server 120; this process can be a pull streaming process, where the media data can be a portion of the data being acquired by the broadcast client 130 that has already been sent to the server 120. Based on this, the server 120 transmits the real-time media data acquired by the broadcast client 130 to the playback client 110, and the playback client 110 renders and plays it to achieve live streaming.

[0048] The aforementioned playback client 110, server 120, and broadcaster client 130 can all be equipped with Web Real-Time Communication (WebRTC) to support real-time audio / video data transmission by web browsers, thereby achieving the aforementioned RTS.

[0049] The aforementioned playback client 110 and broadcaster client 130 can run on any electronic device, including but not limited to: PC (Personal Computer), mobile phone, tablet computer, smart wearable device, computer with wireless transceiver function, VR (virtual reality) terminal device, AR (augmented reality) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc.

[0050] The aforementioned server 120 can be implemented as a regular server, a server cluster, or a cloud server or server cluster. Server 120 can be used to implement some or all of the following services: signaling service, media service, NAT (Network Address Translation) Session Traversal Utilities for NAT (STUN) service, and NAT Traversal Using Relays around NAT (TURN) service. In some embodiments, the above services can be implemented as independent servers, for example, they may include... Figure 1 The signaling server 121, media server 122, and STUN / TURN server 123 shown may include some or all of them. The TURN server can generally implement STUN and TURN services. In other embodiments, some or all of the above services may be integrated into the same server in the form of service units.

[0051] In this embodiment, the signaling service can be used to realize the signaling interaction between the playback client 110 and the server 120; the media service can be used to receive media data sent by the broadcast client 130 and provide media data to the playback client 110; the STUN service is used to detect whether there is NAT around the playback client 110; the TURN service is used to penetrate the NAT around the playback client 110 and realize the establishment of a media data channel between the playback client 110 and the server 120.

[0052] The streaming process of related technologies typically involves establishing a media data channel between the playback client 110 and the server 120, such as establishing a WebRTC media data channel. In this way, the playback client 110 can use WebRTC technology to pull media data such as video data or audio streams from the server 120.

[0053] Reference Figure 2 The diagram illustrates the structure of a client in the related technology, wherein client 201 can be the aforementioned playback client 110. Client 201 can utilize the API (Application Programming Interface) related to WebRTC technology provided by the browser to pull media data such as video data or audio streams from server 202, and decode and render the media data to achieve media data playback.

[0054] Figure 2In this configuration, client 201 may specifically include a peer-to-peer connection module 211 and a video tag module 212. The peer-to-peer connection module 211 can establish a peer-to-peer connection with server 202, receive media data based on the corresponding media channel, and provide media data to the video tag of video tag module 212. Video tag module 212 can embed the video element corresponding to the media data into the HTML (Hypertext Markup Language) page to enable media data playback.

[0055] In practical applications, the video tags in video tag module 212 are based on WebRTC technology. However, the video encoding formats supported by WebRTC technology typically include VP8 (On2 TrueMotion VP8), AV1 (AOMedia Video1), etc., but do not include HEVC format.

[0056] To address the technical problem of limited audio and video encoding formats supported by WebRTC technology in related technologies, this application provides a data processing method. The method specifically includes: acquiring target media data; decoding video data in the target media data using hardware decoding technology or a first network intermediate code; wherein the hardware decoding technology utilizes hardware based on a hardware interface; the first network intermediate code can be intermediate code compiled from a first preset language code with decoding capabilities possessing preset video encoding characteristics, and this first network intermediate code can run in a browser environment; and decoding audio data in the target media data using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code with decoding capabilities possessing preset audio encoding characteristics, and this second network intermediate code can run in a browser environment. Examples of intermediate code may include WASM (WebAssembly), etc.

[0057] This application embodiment utilizes hardware decoding technology or a first network intermediate code to decode video data in the target media data, and utilizes a second network intermediate code to decode audio data in the target media data. Since hardware decoding technology uses hardware to perform decoding based on a hardware interface, this application embodiment can overcome the limitation of video encoding formats supported by WebRTC technology, for example, it can support the HEVC format video encoding type. Because the first network intermediate code can be intermediate code compiled from a first preset language code with decoding capabilities possessing preset video encoding features, this application embodiment can support preset video encoding features not supported by WebRTC technology, such as the HEVC format. Therefore, this application embodiment can increase the applicability of video encoding types.

[0058] This application embodiment utilizes a second network intermediate code to decode the audio data in the target media data. Since this second network intermediate code is compiled from a second preset language code possessing decoding capabilities with preset audio encoding features, this application embodiment can support preset audio encoding features not supported by WebRTC technology, such as the AAC format. Therefore, this application embodiment can expand the applicability of audio encoding types.

[0059] In summary, the embodiments of this application utilize hardware decoding technology, a first network intermediate code, and a second network intermediate code to take over the audio and video decoding part handled by WebRTC technology, which can expand and support more audio and video codec types and encoding characteristics.

[0060] Method Implementation Examples

[0061] Reference Figure 3 The diagram illustrates a flowchart of a data processing method according to an embodiment of this application, which may specifically include the following steps:

[0062] Step 301: Obtain target media data;

[0063] Step 302: Decode the video data in the target media data using hardware decoding technology or the first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding according to the hardware interface; the first network intermediate code can be intermediate code compiled from a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment.

[0064] Step 303: Use the second network intermediate code to decode the audio data in the target media data; the second network intermediate code can be intermediate code compiled from a second preset language code with preset audio encoding characteristics, and the second network intermediate code can run in a browser environment.

[0065] Figure 3 The method embodiment shown includes at least one step that can be performed by a client, which can be the aforementioned playback client 110. Of course, this application embodiment is for... Figure 3 The specific entity that performs the method shown is not restricted.

[0066] In step 301, this embodiment of the application does not limit the communication protocol between the client and the server. For example, the communication protocol between the client and the server may specifically include TCP (Transmission Control Protocol) or UDP (User Datagram Protocol), etc.

[0067] For example, when using TCP, the connection between the client and the server can be a WebSocket connection. Conversely, when using UDP, the connection between the client and the server can be a point-to-point connection using WebRTC technology, and a WebRTC data channel can be used to receive media data from the server.

[0068] In a specific implementation, media data can be obtained from the server based on the data channel, and the target media data can be determined based on the media data obtained from the server; the data channel can be a communication link based on WebRTC (Web Multimedia Communication) technology, and the mode of the data channel can be an unreliable transmission mode.

[0069] WebRTC technology channels can include either data channels or media channels. In related technologies, media channels are used to transmit media data such as audio and video data, while data channels are used to transmit data other than audio and video data, such as command data. In these technologies, media data is used by default for transmission, and media data received via the media channel is by default sent to the video tag module 212; that is, WebRTC technology is used by default to decode the media data received via the media channel.

[0070] In order to overcome the limitation of "using WebRTC technology by default to decode media data received by the media channel", this application embodiment adopts a data channel using WebRTC technology to receive media data from the server.

[0071] WebRTC technology's data channel modes can include reliable transmission mode and unreliable transmission mode. Reliable transmission mode is the default mode for the data channel. It uses an unlimited retransmission mechanism to ensure successful data transmission to the peer; in other words, media data is retransmitted without a limit on the number of retransmissions until it is successfully transmitted to the peer. Unreliable transmission mode uses a limited retransmission mechanism. In other words, media data is retransmitted within a limited number of retransmissions; regardless of whether the media data is successfully transmitted to the peer, retransmission can terminate once the maximum number of retransmissions is reached.

[0072] In this embodiment, the data channel operates in unreliable transmission mode, which reduces the number of retransmissions to some extent, thus improving the transmission efficiency of media data. In this embodiment, the media channel can be in an open state during the process of receiving media data from the server using WebRTC technology.

[0073] In a specific implementation, the process of obtaining media data from the server described above may include: determining the communication protocol corresponding to the received data; and, if the communication protocol is a first communication protocol, using the received data corresponding to the first communication protocol as media data. Received data may refer to data received via a data channel.

[0074] In practical applications, the received data via the data channel can include not only media data but also custom data such as signaling data. This application embodiment can differentiate between different received data based on the communication protocol corresponding to the received data. Specifically, the first communication protocol can correspond to media data, and the second communication protocol can correspond to custom data.

[0075] Those skilled in the art can determine the first communication protocol based on actual application requirements. For example, the first communication protocol can be the UDP protocol to improve the transmission speed of media data. Examples of application layer protocols corresponding to the first communication protocol may include: RTP (Real-time Transport Protocol) or RTCP (Real-time Control Protocol), etc.

[0076] In this embodiment of the application, when the communication protocol is a second communication protocol, the received data corresponding to the second communication protocol can be used as custom data.

[0077] Those skilled in the art can determine a second communication protocol based on actual application requirements. For example, the second communication protocol can be TCP or SCTP (Stream Control Transmission Protocol) to improve the reliability of custom data transmission.

[0078] In one implementation of this application, the process of determining the target media data based on the media data obtained from the server may include: using the media data obtained from the server as the target media data.

[0079] In another implementation of this application, the process of determining the target media data based on the media data obtained from the server may include: performing Quality of Service (QOS) processing on the media data obtained from the server to obtain the target media data.

[0080] In practical implementation, third-party network intermediate code can be used to process the quality of service of the media data obtained from the server in order to obtain the target media data.

[0081] The aforementioned quality of service (QoS) handling can address data transmission issues such as packet loss and jitter in media data, thereby improving the quality of media data transmission. For example, when the communication protocol between the client and server is TCP, or when an unreliable transmission mode using WebRTC technology is employed, data transmission problems such as packet loss and jitter may occur.

[0082] The quality of service (QoS) processing employed in the embodiments of this application may include, but is not limited to, IP (Internet Protocol) priority processing, rate adjustment, packet loss retransmission, forward error correction, and backward error correction. These QoS processing measures can, to a certain extent, control data packet congestion, reduce errors generated during transmission, and improve the transmission quality of media data.

[0083] In practical implementation, JavaScript code or third-party network intermediate code can be used to process media data obtained from the server using Quality of Service (QoS) techniques to obtain the target media data. In practical applications, third-party pre-defined language code with QoS processing capabilities can be written using assembly languages ​​such as C and C++, and then compiled into an intermediate format to generate a third-party network intermediate code with very fast loading and execution speed. This third-party network intermediate code can be intermediate code compiled from the third-party pre-defined language code with QoS processing capabilities, and it can run in a browser environment. Because JavaScript code is interpreted, its execution speed is limited. The third-party network intermediate code, however, can be intermediate code compiled from the third-party pre-defined language code. The third-party pre-defined language code can be the code corresponding to assembly languages ​​such as C and C++.

[0084] In step 302, hardware decoding technology can be used to decode the video data in the target media data. This hardware decoding technology utilizes hardware based on a hardware interface. Therefore, this embodiment can overcome the limitation of audio encoding formats supported by WebRTC technology, for example, it can support HEVC video encoding types. Examples of hardware interfaces may include WebCodec (web coder + decoder), which can be an API provided by a browser. During the decoding of video data in the target media data using WebCodec, the hardware of the electronic device can be invoked, thus improving the decoding performance of the video data.

[0085] This application embodiment can utilize a first network intermediate code to decode video data in the target media data. The first network intermediate code can be intermediate code compiled from a first preset language code possessing decoding capabilities with preset video encoding features, and this first network intermediate code can run in a browser environment. The preset video encoding features can be determined by those skilled in the art based on actual application requirements; for example, preset video encoding features may include HEVC format and B-frames, etc. The first preset language code can be code corresponding to assembly languages ​​such as C and C++.

[0086] Intermediate code can be a type of bytecode, and its corresponding intermediate format can be an encoding format that runs in a browser and can interact with JavaScript.

[0087] In practical applications, a first preset language code can be written using assembly language based on preset video encoding characteristics, and then the first preset language code can be compiled into a first network intermediate code for browser to call.

[0088] The embodiments of this application can perform video decoding in a multi-threaded manner through the first network intermediate code on the browser side, which can improve the video decoding speed.

[0089] Accordingly, the above-mentioned decoding of video data in the target media data using the first network intermediate code includes: the main thread initializing multiple video decoding threads; the main thread sending video decoding tasks corresponding to the video data in the target media data to the idle video decoding threads, so that the idle video decoding threads can use the first network intermediate code to decode the video decoding tasks; wherein, one instance of the first network intermediate code corresponds to one video decoding task.

[0090] Video decoding threads can be browser workers. Workers operate independently of the main thread; after the worker's code completes execution, it returns the result to the main thread. The main thread can assign multiple video decoding tasks and send them to idle video decoding threads. In practice, one video decoding task corresponds to one data packet of video data.

[0091] In this embodiment, one instance of the first network intermediate code can correspond to one video decoding task. Since an instance of the first network intermediate code may not have a built-in thread, cross-domain isolation requirements can be met, thereby improving the security of video decoding. Therefore, this embodiment can be applied to ToB (business-to-business) and other application scenarios with cross-domain isolation requirements.

[0092] The video decoding result of this application embodiment can be image data, and the image data format can be YUV (Luminance Chrominance Chroma), etc. This application embodiment can utilize the image rendering API provided by the browser to draw the image data onto the screen.

[0093] In step 303, the audio data in the target media data can be decoded using the second network intermediate code. This second network intermediate code can be intermediate code compiled from a second preset language code that possesses decoding capabilities with preset audio encoding features.

[0094] The preset audio encoding features can be determined by those skilled in the art based on actual application requirements. For example, preset audio encoding features may include AAC format, etc. The second preset language code may be the code corresponding to assembly languages ​​such as C and C++.

[0095] In practical applications, a second preset language code can be written using assembly language based on preset audio encoding characteristics, and then the second preset language code can be compiled into a second network intermediate code for browsers to call.

[0096] Since the quality of service processing depends to a certain extent on the audio decoding result, and the audio decoding implemented based on the second network intermediate code in this application embodiment can improve the accuracy of the audio decoding result, the quality of service processing in this application embodiment can improve the effect of quality of service processing.

[0097] The embodiments of this application can perform audio decoding in a multi-threaded manner through a second network intermediate code on the browser side, which can improve the audio decoding speed.

[0098] Accordingly, the above-mentioned decoding of audio data in the target media data using the second network intermediate code includes: the main thread initializing multiple audio decoding threads; the main thread sending the audio decoding task corresponding to the audio data in the target media data to the idle audio decoding threads, so that the idle audio decoding threads can use the second network intermediate code to decode the audio decoding task; wherein, one instance of the second network intermediate code corresponds to one audio decoding task.

[0099] The main thread can determine multiple audio decoding tasks and send them to idle audio decoding threads. In practical applications, one audio decoding task can correspond to one data packet of audio data.

[0100] In this embodiment, one instance of the second network intermediate code can correspond to one audio decoding task. Since an instance of the second network intermediate code may not have a built-in thread, cross-domain isolation requirements can be met, thereby improving the security of audio decoding.

[0101] The audio decoding result of this application embodiment can be audio data, and the audio data format can be PCM (Pulse Code Modulation), etc. This application embodiment can utilize the audio rendering API provided by the browser to play the audio data.

[0102] This application's embodiments can be applied to application scenarios such as live streaming, cloud rendering, teleconferencing, video chat, and VoIP, for use in the streaming process within these scenarios. Specifically, the streaming process involves the client retrieving media data stored on the server.

[0103] In summary, the data processing method of this application embodiment utilizes hardware decoding technology or a first network intermediate code to decode video data in target media data, and utilizes a second network intermediate code to decode audio data in the target media data. Since hardware decoding technology uses hardware to perform decoding based on a hardware interface, this application embodiment can overcome the limitation of video encoding formats supported by WebRTC technology, for example, it can support the HEVC format video encoding type. Since the first network intermediate code can be intermediate code compiled from a first preset language code with decoding capabilities possessing preset video encoding features, this application embodiment can support preset video encoding features not supported by WebRTC technology, such as the HEVC format. Therefore, this application embodiment can increase the applicability of video encoding types.

[0104] This application embodiment utilizes a second network intermediate code to decode the audio data in the target media data. Since this second network intermediate code is compiled from a second preset language code possessing decoding capabilities with preset audio encoding features, this application embodiment can support preset audio encoding features not supported by WebRTC technology, such as the AAC format. Therefore, this application embodiment can expand the applicability of audio encoding types.

[0105] In summary, the embodiments of this application utilize hardware decoding technology, a first network intermediate code, and a second network intermediate code to take over the audio and video decoding part handled by WebRTC technology, which can expand and support more audio and video codec types and encoding characteristics.

[0106] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0107] Based on the above embodiments, this application also provides a data processing apparatus, referring to... Figure 4 The diagram shows a structural schematic of a data processing apparatus according to an embodiment of the present application, which may specifically include: an acquisition module 401, a video decoding module 402, and an audio decoding module 403.

[0108] Among them, the acquisition module 401 is used to acquire target media data;

[0109] The video decoding module 402 is used to decode video data in the target media data using hardware decoding technology or a first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding according to the hardware interface; the first network intermediate code is intermediate code compiled according to a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment.

[0110] The audio decoding module 403 is used to decode the audio data in the target media data using the second network intermediate code; the second network intermediate code is intermediate code compiled according to a second preset language code with preset audio encoding characteristics and capable of running in a browser environment.

[0111] Optionally, the acquisition module 401 may specifically include:

[0112] The first acquisition module is used to acquire media data from the server according to the data channel; the data channel is a communication link based on web multimedia communication technology, and the mode of the data channel is unreliable transmission mode.

[0113] The determination module is used to determine the target media data based on the media data obtained from the server.

[0114] Optionally, the first acquisition module may specifically include:

[0115] The communication protocol determination module is used to determine the communication protocol corresponding to the received data;

[0116] The second acquisition module is used to treat the received data corresponding to the first communication protocol as media data when the communication protocol is the first communication protocol.

[0117] Optionally, the first communication protocol can be the UDP protocol.

[0118] Optionally, the device may further include:

[0119] A custom data acquisition module is used to treat the received data corresponding to the second communication protocol as custom data when the communication protocol is the second communication protocol.

[0120] Optionally, the audio decoding module 403 may include:

[0121] The initialization module is used to initialize multiple audio decoding threads using the main thread.

[0122] The task distribution module is used to send the audio decoding task corresponding to the audio data in the target media data to the audio decoding thread in the idle state using the main thread, so that the audio decoding thread in the idle state can decode the audio decoding task using the second network intermediate code; wherein, one instance of the second network intermediate code corresponds to one audio decoding task.

[0123] Optionally, the determining module may include:

[0124] The Quality of Service (QoS) processing module is used to process media data obtained from the server using third-party network intermediate code to obtain target media data. The third-party network intermediate code is intermediate code compiled from a third preset language code with QoS processing capabilities, and the third-party network intermediate code can run in a browser environment.

[0125] In summary, the data processing apparatus of this application embodiment decodes video data in target media data using hardware decoding technology or a first network intermediate code, and decodes audio data in the target media data using a second network intermediate code. Since hardware decoding technology uses hardware to perform decoding based on a hardware interface, this application embodiment can overcome the limitation of video encoding formats supported by WebRTC technology, for example, it can support the HEVC format video encoding type. Since the first network intermediate code can be intermediate code compiled from a first preset language code with decoding capabilities possessing preset video encoding features, this application embodiment can support preset video encoding features not supported by WebRTC technology, such as the HEVC format. Therefore, this application embodiment can increase the applicability of video encoding types.

[0126] This application embodiment utilizes a second network intermediate code to decode the audio data in the target media data. Since this second network intermediate code is compiled from a second preset language code possessing decoding capabilities with preset audio encoding features, this application embodiment can support preset audio encoding features not supported by WebRTC technology, such as the AAC format. Therefore, this application embodiment can expand the applicability of audio encoding types.

[0127] In summary, the embodiments of this application utilize hardware decoding technology, a first network intermediate code, and a second network intermediate code to take over the audio and video decoding part handled by WebRTC technology, which can expand and support more audio and video codec types and encoding characteristics.

[0128] Reference Figure 5 The diagram shows a structural schematic of a data processing apparatus according to an embodiment of this application, which may specifically include: a point-to-point connection module 501, a parsing and distribution module 502, a custom data processing module 503, a quality of service processing module 504, a video decoding module 505, and an audio decoding module 506.

[0129] Among them, the point-to-point connection module 501 is used to establish a point-to-point connection with the server and obtain received data based on the corresponding data channel.

[0130] The parsing and distribution module 502 is used to determine the communication protocol corresponding to the received data and distribute the received data according to the communication protocol. Specifically, if the communication protocol is a first communication protocol, the received data corresponding to the first communication protocol is distributed as media data to the quality of service processing module 504; or, if the communication protocol is a second communication protocol, the received data corresponding to the first communication protocol is distributed as custom data to the custom data processing module 503.

[0131] The custom data processing module 503 is used to process custom data such as signaling data.

[0132] The Quality of Service (QoS) processing module 504 is used to process the media data obtained from the server to obtain the target media data. Specifically, QoS processing of the media data can be performed using JavaScript code or third-party network intermediate code.

[0133] The video decoding module 505 is used to decode the video data in the target media data using hardware decoding technology or first network intermediate code to obtain YUV format image data.

[0134] The audio decoding module 506 is used to decode the audio data in the target media data using the intermediate code of the second network to obtain audio data in PCM format.

[0135] The video decoding result of this application embodiment can be image data, and the image data format can be YUV, etc. This application embodiment can utilize the image rendering API provided by the browser to draw the image data onto the screen.

[0136] The audio decoding result of this application embodiment can be audio data, and the audio data format can be PCM, etc. This application embodiment can utilize the audio rendering API provided by the browser to play the audio data.

[0137] Reference Figure 6 The diagram illustrates the structure of a data processing system according to an embodiment of this application, which may specifically include: a server 601 and a client 602; wherein, the client 602 may specifically include: a point-to-point connection module 621, a parsing and distribution module 622, a custom data processing module 623, a quality of service processing module 624, a video decoding module 625, and an audio decoding module 626.

[0138] Among them, the point-to-point connection module 621 is used to establish a point-to-point connection with the server and obtain received data based on the corresponding data channel.

[0139] The parsing and distribution module 622 is used to determine the communication protocol corresponding to the received data and distribute the received data according to the communication protocol. Specifically, if the communication protocol is a first communication protocol, the received data corresponding to the first communication protocol is distributed as media data to the quality of service processing module 624; or, if the communication protocol is a second communication protocol, the received data corresponding to the first communication protocol is distributed as custom data to the custom data processing module 623.

[0140] The custom data processing module 623 is used to process custom data such as signaling data.

[0141] The Quality of Service (QoS) processing module 624 is used to perform QoS processing on the media data obtained from the server before the video decoding module 625 decodes the video data in the media data and the audio decoding module 626 decodes the audio data in the media data, in order to obtain the target media data. The third network intermediate code is intermediate code compiled from a third preset language code with QoS processing capabilities, and the third network intermediate code can run in a browser environment. Specifically, QoS processing of the media data can be performed using JS code or the third network intermediate code.

[0142] The video decoding module 625 is used to decode the video data in the target media data using hardware decoding technology or first network intermediate code to obtain YUV format image data.

[0143] The audio decoding module 626 is used to decode the audio data in the target media data using the intermediate code of the second network to obtain audio data in PCM format.

[0144] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0145] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes devices such as servers and terminal devices.

[0146] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired, and the apparatus may include electronic devices such as servers (clusters) and terminals. Figure 7 An exemplary apparatus 1700 is schematically shown that can be used to implement the various embodiments described in this application.

[0147] In one embodiment, Figure 7 An exemplary device 1700 is shown, which includes one or more processors 1702, a control module (chipset) 1704 coupled to at least one of the processors 1702, a memory 1706 coupled to the control module 1704, a non-volatile memory (NVM) / storage device 1708 coupled to the control module 1704, one or more input / output devices 1710 coupled to the control module 1704, and a network interface 1712 coupled to the control module 1704.

[0148] Processor 1702 may include one or more single-core or multi-core processors, and processor 1702 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1700 can serve as a server, terminal, or other device as described in the embodiments of this application.

[0149] In some embodiments, apparatus 1700 may include one or more computer-readable media (e.g., memory 1706 or NVM / storage device 1708) having instructions 1714 and one or more processors 1702 that are combined with the one or more computer-readable media and configured to execute instructions 1714 to implement a module thereby performing the actions described in this disclosure.

[0150] In one embodiment, the control module 1704 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1702 and / or any suitable device or component communicating with the control module 1704.

[0151] The control module 1704 may include a memory controller module to provide an interface to the memory 1706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0152] Memory 1706 may be used, for example, to load and store data and / or instructions 1714 for device 1700. In one embodiment, memory 1706 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1706 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0153] In one embodiment, control module 1704 may include one or more input / output controllers to provide interfaces to NVM / storage device 1708 and (one or more) input / output devices 1710.

[0154] For example, NVM / storage device 1708 may be used to store data and / or instructions 1714. NVM / storage device 1708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0155] NVM / storage device 1708 may include storage resources that are part of a device on which device 1700 is mounted, or that are accessible by the device but do not necessarily have to be part of the device. For example, NVM / storage device 1708 may be accessed via a network via one or more input / output devices 1710.

[0156] One or more input / output devices 1710 may provide an interface for device 1700 to communicate with any other suitable device. Input / output devices 1710 may include communication components, audio components, sensor components, etc. Network interface 1712 may provide an interface for device 1700 to communicate via one or more networks. Device 1700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0157] In one embodiment, at least one of the processors 1702 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1704. In one embodiment, at least one of the processors 1702 may be logically packaged with one or more controllers of the control module 1704 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1702 may be integrated with the logic of one or more controllers of the control module 1704 on the same die. In one embodiment, at least one of the processors 1702 may be integrated with the logic of one or more controllers of the control module 1704 on the same die to form a system-on-a-chip (SoC).

[0158] In various embodiments, device 1700 may be, but is not limited to, a terminal device such as a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1700 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0159] In the device 1700, a main control chip can be used as a processor or control module, sensor data, position information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0160] This application also provides an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in this application.

[0161] This application also provides one or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform one or more of the methods described in this application.

[0162] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0164] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0167] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0169] The above provides a detailed description of a data processing method, a data processing device, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, Applied to a client, the method includes: Acquire target media data; the connection between the client and the server is a point-to-point connection using WebRTC technology. The client receives media data from the server through a WebRTC data channel and determines the target media data based on the media data acquired from the server. The video data in the target media data is decoded using hardware decoding technology or a first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding based on a hardware interface; the first network intermediate code is intermediate code compiled from a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment. The audio data in the target media data is decoded using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code with decoding capabilities based on preset audio encoding features, and the second network intermediate code can run in a browser environment.

2. The method according to claim 1, characterized in that, The data channel operates in an unreliable transmission mode.

3. The method according to claim 2, characterized in that, The process of obtaining media data from the server includes: Determine the communication protocol corresponding to the received data; When the communication protocol is the first communication protocol, the received data corresponding to the first communication protocol is used as media data.

4. The method according to claim 3, characterized in that, The first communication protocol is the UDP protocol.

5. The method according to claim 3, characterized in that, The method further includes: When the communication protocol is the second communication protocol, the received data corresponding to the second communication protocol is used as custom data.

6. The method according to any one of claims 1 to 5, characterized in that, The step of decoding the audio data in the target media data using the second network intermediate code includes: The main thread initializes multiple audio decoding threads; The main thread sends the audio decoding task corresponding to the audio data in the target media data to the idle audio decoding thread, so that the idle audio decoding thread can use the second network intermediate code to decode the audio decoding task; wherein, one instance of the second network intermediate code corresponds to one audio decoding task.

7. The method according to claim 2, characterized in that, The step of determining the target media data based on the media data obtained from the server includes: The media data obtained from the server is processed using third-party network intermediate code to obtain the target media data. The third-party network intermediate code is intermediate code compiled from a third preset language code with quality of service processing capabilities, and the third-party network intermediate code can run in a browser environment.

8. A data processing apparatus, characterized in that, Applied to a client, the device includes: The acquisition module is used to acquire target media data; the connection between the client and the server is a point-to-point connection using WebRTC technology. The client receives media data from the server using the WebRTC data channel and determines the target media data based on the media data acquired from the server. The video decoding module is used to decode video data in the target media data using hardware decoding technology or a first network intermediate code; wherein, the hardware decoding technology uses hardware to perform decoding according to a hardware interface; the first network intermediate code is intermediate code compiled from a first preset language code with preset video encoding characteristics, and the first network intermediate code can run in a browser environment. The audio decoding module is used to decode the audio data in the target media data using a second network intermediate code; the second network intermediate code is intermediate code compiled from a second preset language code with preset audio encoding characteristics and capable of running in a browser environment.

9. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1-7.

10. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method as described in any one of claims 1-7.

Images