Data transmission method and device based on WebRTC

Abstract

The embodiment of the invention relates to a data transmission method and device based on WebRTC. The method comprises the steps that target data to be transmitted and hardware characteristics of current electronic equipment are acquired; determining a target coding type and a target decoding type based on the target data; screening a target encoder and a target decoder in a self-defined WebRTC hardware video encoding and decoding factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic equipment; performing encoding processing on the target data based on a target encoder to obtain encoded target data; and transmitting the coded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder. Therefore, different hardware and operating system versions can be adapted, the problem of compatibility between devices is solved, and the efficiency and quality of data transmission are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a data transmission method and device based on WebRTC. Background Art

[0002] Web Real-Time Communications (WebRTC) is a real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary to achieve the transmission of video streams and / or audio streams or any other data. As the main means of achieving real-time communication, WebRTC technology is widely used in video conferencing, real-time chat and other fields. Therefore, the high requirements of WebRTC technology for video encoding make the encoding and decoding process the key to optimization.

[0003] However, on the Android platform, due to severe device fragmentation, different devices have different levels of support for video codecs such as H264, making it difficult to achieve efficient and stable video communication. Summary of the invention

[0004] In view of this, in order to solve the above technical problems or part of the technical problems, an embodiment of the present invention provides a data transmission method and device based on WebRTC.

[0005] In a first aspect, an embodiment of the present invention provides a data transmission method based on WebRTC, including:

[0006] Acquire target data to be transmitted and hardware characteristics of the current electronic device;

[0007] Determining a target encoding type and a target decoding type based on the target data;

[0008] In a custom WebRTC hardware video codec factory, a target encoder and a target decoder are selected based on the target encoding type, the target decoding type, and the hardware characteristics of the current electronic device;

[0009] Performing encoding processing on the target data based on the target encoder to obtain encoded target data;

[0010] The encoded target data is transmitted to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

[0011] In a possible implementation, the method further includes:

[0012] Acquire target data to be transmitted based on a data acquisition device and determine a target data type corresponding to the target data, wherein the target data type at least includes a video data type;

[0013] The hardware characteristics of the current electronic device are obtained and it is determined whether the hardware characteristics support hardware encoding, wherein the hardware characteristics at least include a system version, a chip type, and a device model.

[0014] In a possible implementation, the method further includes:

[0015] If the target data type is a video data type, the target encoding type is determined to be video encoding and the target decoding type is determined to be video decoding.

[0016] In a possible implementation, the method further includes:

[0017] Filter the first target encoder that supports the target encoding type and the first target decoder that supports the target decoding type through the MediaCodecUtils.codecSupportsType method in the custom WebRTC hardware video codec factory;

[0018] Filter, by the MediaCodecUtils.selectColorFormat method, a second target encoder and a second target decoder that support the color format corresponding to the target data in the first target encoder and the first target decoder;

[0019] A target encoder and a target decoder supporting hardware characteristics of the current electronic device are screened in the second target encoder and the second target decoder based on a user-defined function.

[0020] In a possible implementation, the method further includes:

[0021] The target encoder and target decoder are dynamically configured based on the hardware characteristics of the current electronic device through the createEncoder method.

[0022] In a possible implementation, the method further includes:

[0023] Extract frame data from the target data through VideoFrame and VideoFrame.Buffer;

[0024] The extracted frame data is input into the target encoder for encoding processing to obtain encoded target data.

[0025] In a possible implementation, the method further includes:

[0026] Based on the SDK version of the current electronic device, support for a specific video encoding type is dynamically enabled when configuring the target encoder, wherein the specific video encoding type includes at least VP8 and VP9 types.

[0027] In a second aspect, an embodiment of the present invention provides a data transmission device based on WebRTC, including:

[0028] An acquisition module, used to acquire target data to be transmitted and hardware characteristics of the current electronic device;

[0029] A determination module, configured to determine a target encoding type and a target decoding type based on the target data;

[0030] A screening module, used for screening a target encoder and a target decoder in a custom WebRTC hardware video codec factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device;

[0031] An encoding module, used for encoding the target data based on the target encoder to obtain encoded target data;

[0032] The transmission module is used to transmit the encoded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a WebRTC-based data transmission program stored in the memory to implement the WebRTC-based data transmission method described in the first aspect above.

[0034] In a fourth aspect, an embodiment of the present invention provides a storage medium, including: the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the WebRTC-based data transmission method described in the first aspect above.

[0035] The data transmission scheme based on WebRTC provided by the embodiment of the present invention obtains the target data to be transmitted and the hardware characteristics of the current electronic device; determines the target encoding type and the target decoding type based on the target data; selects the target encoder and the target decoder based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device in the custom WebRTC hardware video codec factory; encodes the target data based on the target encoder to obtain the encoded target data; transmits the encoded target data to the data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder. According to this scheme, the official general interface (VideoEncoderFactory) is used to implement a custom encoder factory for a specific video format, covering the selection and configuration of the codec, etc., which can adapt to different hardware and operating system versions, solves the compatibility problem between devices and improves the efficiency and quality of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a data transmission method based on WebRTC provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart of another WebRTC-based data transmission method provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a WebRTC sending end video processing flow provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a WebRTC receiving end video processing flow provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a WebRTC video encoder selection process provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the structure of a WebRTC-based data transmission device provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0045] Figure 1 A flow chart of a data transmission method based on WebRTC provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method specifically includes:

[0046] S11, obtaining target data to be transmitted and hardware characteristics of the current electronic device.

[0047] In the embodiment of the present invention, the data type of the target data includes a video data type, an audio data type, an audio and video data type or other data types. The current electronic device may be an Android phone, and the hardware characteristics include at least a system version, a chip type and a device model.

[0048] The embodiment of the present invention provides a method for data transmission based on a custom WebRTC hardware video codec factory (CustomHardwareVideoEncoderFactory) on an Android terminal, aiming to solve the compatibility problem between devices and improve the efficiency and quality of video communication. The method uses the official universal interface (VideoEncoderFactory) to implement a custom encoder factory for the H264 video format. The method covers the selection and configuration of the codec to adapt to different hardware and operating system versions.

[0049] S12. Determine a target encoding type and a target decoding type based on the target data.

[0050] S13. In the customized WebRTC hardware video codec factory, a target encoder and a target decoder are selected based on the target encoding type, the target decoding type, and the hardware characteristics of the current electronic device.

[0051] In the custom WebRTC hardware video codec factory, an advanced encoder detection and selection mechanism is implemented. This mechanism is based on inheriting and implementing the VideoEncoderFactory and VideoDecoderFactory interfaces. Through the methods createEncoder and createDecoder, the system can perform codec screening and matching based on the input VideoCodecInfo parameters and the specific hardware characteristics of the current Android device.

[0052] S14. Encode the target data based on the target encoder to obtain encoded target data.

[0053] After the target encoder and target decoder are selected, the createEncoder method will dynamically configure the target encoder and target decoder based on device-specific conditions (such as hardware capabilities and operating system version).

[0054] Optionally, based on the Android SDK version and hardware features supported by the current electronic device, support for VP8 and VP9 video encoding can be dynamically enabled when creating the target encoder to enhance compatibility across platforms and older devices.

[0055] Use WebRTC's built-in functions to add data sources to the PeerConnection manager, and the WebRTC manager will subsequently manage data encoding and transmission. Use the target encoder created by the custom encoding factory, and call the encode method by WebRTC to encode the data and obtain the encoded target data.

[0056] S15, transmitting the encoded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

[0057] WebRTC uses RTP packaging to transmit target data (sending mode), pushes the locally created data stream to the RTC server, and sends the local data stream Session Description Protocol (SDP) information to the call receiver. The other party can obtain the data stream pushed by the sender through SDP. The use process of SDP in WebRTC is as follows:

[0058] Create Offer: The party initiating the WebRTC session (called Offerer) will first create an SDP Offer message to describe its own media capabilities and network information, and send it to the recipient (called Answerer) through the signaling server.

[0059] Create Answer: After receiving the Offer, the Answerer parses the SDP information in it and negotiates based on its own capabilities. Finally, it creates an SDP Answer message that describes the media format and network information it agrees to, and sends it back to the Offerer.

[0060] Establishing a connection: After receiving the Answer, the Offerer parses the SDP information in it and establishes a media streaming transmission connection based on the negotiation results.

[0061] Furthermore, the encoded target data is transmitted to a data receiving end, so that the data receiving end decodes and displays the received target data based on a target decoder.

[0062] The data transmission method based on WebRTC provided in the embodiment of the present invention obtains the target data to be transmitted and the hardware characteristics of the current electronic device; determines the target encoding type and the target decoding type based on the target data; selects the target encoder and the target decoder based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device in the custom WebRTC hardware video codec factory; encodes the target data based on the target encoder to obtain the encoded target data; transmits the encoded target data to the data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder. According to this method, the official general interface (VideoEncoderFactory) is used to implement a custom encoder factory for a specific video format, covering the selection and configuration of the codec, etc., and can adapt to different hardware and operating system versions, solve the compatibility problem between devices and improve the efficiency and quality of data transmission.

[0063] Figure 2 A flowchart of another WebRTC-based data transmission method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method specifically includes:

[0064] S21. Acquire target data to be transmitted based on a data acquisition device and determine a target data type corresponding to the target data.

[0065] In the embodiment of the present invention, taking the data type of the target data as the video data type as an example, Figure 3 Schematic diagram of the video processing flow of the WebRTC sender, Figure 4 WebRTC receiver video processing flow diagram and Figure 5 The WebRTC video encoder selection process diagram is described in detail:

[0066] First, create a video source, capture video data based on the data acquisition device (for example, a camera or screen) on the current electronic device, and provide it to WebRTC for processing.

[0067] Call the createVideoSource() method of the PeerConnectionFactory object and pass in a Boolean parameter this.mVideoCapturer.isScreencast().

[0068] this.mVideoCapturer.isScreencast() is used to determine whether mVideoCapturer is currently in screen capture mode. Returning true means that screen sharing is in progress, and false means that camera video is being captured.

[0069] Android cameras capture video frames. These frames are usually raw image data in YUV format.

[0070] By getting the VideoFrame.Buffer in the VideoFrame, and then extracting the frame data from the buffer (converted to VideoFrame.I420Buffer, the original YUV format image is not convenient to convert).

[0071] S22: Acquire hardware characteristics of the current electronic device and determine whether the hardware characteristics support hardware encoding.

[0072] Check the hardware characteristics of the current electronic device, where the hardware characteristics include at least the system version, chip type and device model. Provide a differentiated hardware device exclusion list (customization is supported), for example: "SAMSUNG-SGH-I337", "Nexus7", "Nexus4" and other device hardware have poor bit rate control for H.264 encoding. Provide a general hardware encoder matching mechanism, such as whether the encoder name prefix is ​​related to the following chip manufacturers, for example: Qualcomm (OMX.qcom.), Samsung Exynos (OMX.Exynos.), Google (OMX.google.) and other devices support H264 encoding on the corresponding Android version.

[0073] S23. If the target data type is a video data type, determine that the target encoding type is video encoding and the target decoding type is video decoding.

[0074] S24. Filter the first target encoder that supports the target encoding type and the first target decoder that supports the target decoding type through the MediaCodecUtils.codecSupportsType method in the custom WebRTC hardware video codec factory.

[0075] S25. Filter, by using the MediaCodecUtils.selectColorFormat method, a second target encoder and a second target decoder that support the color format corresponding to the target data in the first target encoder and the first target decoder.

[0076] S26: Filter, based on the user-defined function, a target encoder and a target decoder that support hardware characteristics of the current electronic device from among the second target encoder and the second target decoder.

[0077] The following is a unified description of S23-S26:

[0078] Encoder / Decoder Compatibility Check:

[0079] Use MediaCodecList to traverse all available encoders on the current electronic device. Use the MediaCodecUtils.codecSupportsType method to check whether the encoder supports the specified encoding type; use the MediaCodecUtils.selectColorFormat method to check whether the encoder supports the specified color format; isHardwareSupportedInCurrentSdk(info,type) to check whether the encoder supports hardware encoding, and check whether the current Android version (add version restrictions according to the Android hardware API system version requirements) meets the requirements.

[0080] MediaCodecList: A class used to retrieve and manage media codecs on the Android platform. It allows applications to query the codecs available on the device and obtain detailed information about the codecs, such as supported codec types, color formats, hardware acceleration support, etc.

[0081] MediaCodecUtils.codecSupportsType: A utility method for checking whether a MediaCodecInfo object supports a given encoding or decoding type.

[0082] MediaCodecUtils.selectColorFormat: A utility method used to select the best color format from multiple color format options. It is mainly used to check whether the MediaCodecInfo (current encoder information) object supports at least one color format defined in the predefined encoder color format list. This check ensures that the encoder can encode in an acceptable color format, thereby ensuring the normal encoding.

[0083] isHardwareSupportedInCurrentSdk(info,type): Depending on the encoding type, different judgment methods are called to check whether the hardware module is supported. These judgment methods may contain judgment logic based on conditions such as Android version, chip type, device model, etc. to ensure that the hardware encoding is feasible on the current device and SDK version.

[0084] The target encoder and target decoder can be screened and determined by the above method.

[0085] S27. Dynamically configure the target encoder and target decoder based on the hardware characteristics of the current electronic device through the createEncoder method.

[0086] After selecting the encoder, the createEncoder method is used to dynamically configure the target encoder and decoder based on the specific conditions of the current electronic device (such as hardware capabilities and operating system version). Key configuration items include:

[0087] Color format selection: Select the optimal color format based on the capabilities of the target encoder and decoder to improve encoding efficiency and reduce energy consumption.

[0088] Color format enumeration: MediaCodecUtils.TEXTURE_COLOR_FORMATS represents a set of predefined color formats supported by Surface textures, such as COLOR_FormatSurface.

[0089] Encoder capability query: info.getCapabilitiesForType(mime) is used to obtain the encoder capability information of the specified MIME type, which includes the color formats supported by the encoder.

[0090] Color format matching: The MediaCodecUtils.selectColorFormat() method iterates over each color format in TEXTURE_COLOR_FORMATS and checks whether the encoder supports the format. If a matching format is found, it is assigned to the surfaceColorFormat variable, otherwise surfaceColorFormat is null.

[0091] Bitrate adjuster selection: Select the encoding bitrate adjuster according to the hardware performance of the current electronic device to balance the video quality and transmission rate. Specific implementation: Define a private method named createBitrateAdjuster to create a suitable bitrate adjuster based on the video encoding type and the target encoder name.

[0092] Determine the encoder name: The code checks whether the encoder name codecName starts with "OMX.Exynos." This indicates that the encoder may be the hardware encoder used on the Samsung Exynos processor.

[0093] For Exynos encoder: If it is an Exynos encoder, select different bitrate adjusters according to the video encoding type. If it is VP8 encoding, create a DynamicBitrateAdjuster object. Otherwise (such as H.264 encoding), create a FramerateBitrateAdjuster object.

[0094] Other encoders: If it is not an Exynos encoder, create a default BaseBitrateAdjuster object.

[0095] Optionally, based on the Android SDK version and hardware features supported by the current electronic device, dynamically enable support for VP8 and VP9 video encoding when creating the encoder to enhance compatibility across platforms and older devices.

[0096] Specifically, select the appropriate judgment method according to the encoding type: the isHardwareSupportedInCurrentSdk method calls the corresponding judgment method according to the incoming video encoding type (VP8, VP9 or H264), such as isHardwareSupportedInCurrentSdkVp8, isHardwareSupportedInCurrentSdkVp9 or isHardwareSupportedInCurrentSdkH264.

[0097] Determine hardware support based on target encoder name and SDK version: Each determination method for a specific encoding type checks the name of the target encoder and the SDK version of the current electronic device to determine whether the current environment supports hardware encoding of that encoding type.

[0098] Handling special cases: some special processing logic for encoders and device models of specific manufacturers, such as encoders for manufacturers such as Samsung, Qualcomm, and Intel, as well as some device models that are known to be unable to perform hardware encoding normally.

[0099] Provide callback mechanism: If the preset judgment logic in the code cannot meet the needs, developers can also implement customized judgment logic by setting the videoEncoderSupportedCallback callback function.

[0100] S28. Extract frame data from the target data through VideoFrame and VideoFrame.Buffer.

[0101] By getting the VideoFrame.Buffer in the VideoFrame, and then extracting the frame data from the buffer (converted to VideoFrame.I420Buffer (buffer) before extracting, the original YUV format image is not convenient for conversion).

[0102] S29, inputting the extracted frame data into the target encoder for encoding processing to obtain encoded target data.

[0103] Use the WebRTC built-in function to add the created video source (Video Source) to the PeerConnection manager, and the WebRTC manager will subsequently manage video encoding and transmission.

[0104] By using the encoder (HardwareVideoEncoder) created by the custom encoding factory, WebRTC calls the encode method to process the video frame.

[0105] Furthermore, WebRTC transmits video data (sending mode) using RTP packaging, pushes the locally created video stream to the RTC server, and sends the local video stream SDP (Session Description Protocol) information to the other party of the call. The other party can obtain the video stream pushed by us through SDP. The use process of SDP in WebRTC is as follows:

[0106] Create Offer: The party initiating the WebRTC session (called Offerer) will first create an SDP Offer message, describe its own media capabilities and network information, and send it to the recipient (called Answerer) through the signaling server.

[0107] Create Answer: After receiving the Offer, the Answerer will parse the SDP information in it and negotiate according to its own capabilities. Finally, it will create an SDP Answer message, describe the media format and network information it agrees to, and send it back to the Offerer.

[0108] Establishing a connection: After receiving the Answer, the Offerer parses the SDP information in it and establishes a media streaming transmission connection based on the negotiation results.

[0109] Furthermore, the encoded target data is transmitted to a data receiving end, so that the data receiving end decodes and displays the received target data based on a target decoder.

[0110] The embodiments of the present invention mainly focus on realizing a flexible and efficient custom WebRTC video codec factory on the Android side, which can adapt to a wide range of device types and hardware configurations. In particular, it shows significant advantages in improving the support for H264 and the performance adjustment of multiple codecs, and effectively addresses the compatibility and performance challenges brought about by device fragmentation. The custom video codec factory method makes the WebRTC implementation on Android devices more efficient and stable, especially in the face of multiple devices and different network environments, it can achieve better communication effects. This technology has important market potential and broad application prospects, especially in the fields of video conferencing, distance education, real-time monitoring, etc., which can provide significant performance improvements.

[0111] The data transmission method based on WebRTC provided in the embodiment of the present invention obtains the target data to be transmitted and the hardware characteristics of the current electronic device; determines the target encoding type and the target decoding type based on the target data; selects the target encoder and the target decoder based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device in the custom WebRTC hardware video codec factory; encodes the target data based on the target encoder to obtain the encoded target data; transmits the encoded target data to the data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder. According to this method, the official general interface (VideoEncoderFactory) is used to implement a custom encoder factory for a specific video format, covering the selection and configuration of the codec, etc., and can adapt to different hardware and operating system versions, solve the compatibility problem between devices and improve the efficiency and quality of data transmission.

[0112] Figure 6 A schematic diagram of a data transmission device based on WebRTC provided by an embodiment of the present invention specifically includes:

[0113] The acquisition module 601 is used to acquire the target data to be transmitted and the hardware characteristics of the current electronic device. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0114] The determination module 602 is used to determine the target encoding type and the target decoding type based on the target data. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0115] The screening module 603 is used to screen the target encoder and the target decoder in the customized WebRTC hardware video codec factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0116] The encoding module 604 is used to encode the target data based on the target encoder to obtain the encoded target data. For detailed description, please refer to the relevant description of the above method embodiment, which will not be repeated here.

[0117] The transmission module 605 is used to transmit the encoded target data to the data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0118] The WebRTC-based data transmission device provided in this embodiment may be as follows: Figure 6 The WebRTC-based data transmission device shown in FIG. 1 may perform the following steps: Figure 1-2 All steps of the WebRTC-based data transmission method are implemented Figure 1-2 The technical effect of the data transmission method based on WebRTC is shown in the figure. For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0119] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 7 The electronic device 700 shown includes: at least one processor 701, a memory 702, at least one network interface 704 and other user interfaces 703. The various components in the electronic device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 705 is not described in detail. Figure 7 Various buses are labeled as bus system 705.

[0120] The user interface 703 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0121] It can be understood that the memory 702 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0122] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 7021 and an application 7022 .

[0123] The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 7022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 7022.

[0124] In the embodiment of the present invention, by calling the program or instruction stored in the memory 702, specifically, the program or instruction stored in the application 7022, the processor 701 is used to execute the method steps provided by each method embodiment, for example, including:

[0125] Acquire target data to be transmitted and hardware characteristics of the current electronic device; determine a target encoding type and a target decoding type based on the target data; select a target encoder and a target decoder in a custom WebRTC hardware video codec factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device; encode the target data based on the target encoder to obtain encoded target data; transmit the encoded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

[0126] In one possible implementation, based on a data acquisition device, target data to be transmitted is acquired and a target data type corresponding to the target data is determined, wherein the target data type includes at least a video data type; hardware characteristics of the current electronic device are acquired and it is determined whether the hardware characteristics support hardware encoding, wherein the hardware characteristics include at least a system version, a chip type, and a device model.

[0127] In a possible implementation, if the target data type is a video data type, the target encoding type is determined to be video encoding and the target decoding type is determined to be video decoding.

[0128] In one possible implementation, a first target encoder supporting the target encoding type and a first target decoder supporting the target decoding type are screened through the MediaCodecUtils.codecSupportsType method in a custom WebRTC hardware video codec factory; a second target encoder and a second target decoder supporting the color format corresponding to the target data are screened in the first target encoder and the first target decoder through the MediaCodecUtils.selectColorFormat method; and a target encoder and a target decoder supporting the hardware characteristics of the current electronic device are screened in the second target encoder and the second target decoder based on a custom function.

[0129] In a possible implementation, the target encoder and target decoder are dynamically configured based on the hardware characteristics of the current electronic device through a createEncoder method.

[0130] In a possible implementation, frame data is extracted from the target data through VideoFrame and VideoFrame.Buffer; the extracted frame data is input into the target encoder for encoding processing to obtain encoded target data.

[0131] In a possible implementation, based on the SDK version of the current electronic device, support for a specific video encoding type is dynamically enabled when configuring the target encoder, wherein the specific video encoding type includes at least VP8 and VP9 types.

[0132] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with its hardware.

[0133] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0134] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0135] The electronic device provided in this embodiment may be Figure 7 The electronic device shown in FIG. 1 may perform the following steps: Figure 1-2 All steps of the WebRTC-based data transmission method are implemented Figure 1-2 The technical effect of the data transmission method based on WebRTC is shown in the figure. For details, please refer to Figure 1-2 For the sake of brevity, the relevant description is not repeated here.

[0136] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0137] When one or more programs in the storage medium can be executed by one or more processors, the WebRTC-based data transmission method executed on the electronic device side can be implemented.

[0138] The processor is used to execute the WebRTC-based data transmission program stored in the memory to implement the following steps of the WebRTC-based data transmission method performed on the electronic device side:

[0139] Acquire target data to be transmitted and hardware characteristics of the current electronic device; determine a target encoding type and a target decoding type based on the target data; select a target encoder and a target decoder in a custom WebRTC hardware video codec factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device; encode the target data based on the target encoder to obtain encoded target data; transmit the encoded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

[0140] In one possible implementation, based on a data acquisition device, target data to be transmitted is acquired and a target data type corresponding to the target data is determined, wherein the target data type includes at least a video data type; hardware characteristics of the current electronic device are acquired and it is determined whether the hardware characteristics support hardware encoding, wherein the hardware characteristics include at least a system version, a chip type, and a device model.

[0141] In a possible implementation, if the target data type is a video data type, the target encoding type is determined to be video encoding and the target decoding type is determined to be video decoding.

[0142] In one possible implementation, a first target encoder supporting the target encoding type and a first target decoder supporting the target decoding type are screened through the MediaCodecUtils.codecSupportsType method in a custom WebRTC hardware video codec factory; a second target encoder and a second target decoder supporting the color format corresponding to the target data are screened in the first target encoder and the first target decoder through the MediaCodecUtils.selectColorFormat method; and a target encoder and a target decoder supporting the hardware characteristics of the current electronic device are screened in the second target encoder and the second target decoder based on a custom function.

[0143] In a possible implementation, the target encoder and target decoder are dynamically configured based on the hardware characteristics of the current electronic device through a createEncoder method.

[0144] In a possible implementation, frame data is extracted from the target data through VideoFrame and VideoFrame.Buffer; the extracted frame data is input into the target encoder for encoding processing to obtain encoded target data.

[0145] In a possible implementation, based on the SDK version of the current electronic device, support for a specific video encoding type is dynamically enabled when configuring the target encoder, wherein the specific video encoding type includes at least VP8 and VP9 types.

[0146] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0148] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data transmission method based on WebRTC, characterized in that: include: Acquire target data to be transmitted and hardware characteristics of the current electronic device; Determining a target encoding type and a target decoding type based on the target data; In a custom WebRTC hardware video codec factory, a target encoder and a target decoder are selected based on the target encoding type, the target decoding type, and the hardware characteristics of the current electronic device; Performing encoding processing on the target data based on the target encoder to obtain encoded target data; The encoded target data is transmitted to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

2. The method according to claim 1, characterized in that The step of obtaining the target data to be transmitted and the hardware characteristics of the current electronic device includes: Acquire target data to be transmitted based on a data acquisition device and determine a target data type corresponding to the target data, wherein the target data type at least includes a video data type; The hardware characteristics of the current electronic device are obtained and it is determined whether the hardware characteristics support hardware encoding, wherein the hardware characteristics at least include a system version, a chip type, and a device model.

3. The method according to claim 2, characterized in that The determining of a target encoding type and a target decoding type based on the target data comprises: If the target data type is a video data type, the target encoding type is determined to be video encoding and the target decoding type is determined to be video decoding.

4. The method according to claim 3, characterized in that The step of selecting a target encoder and a target decoder based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device in the customized WebRTC hardware video codec factory includes: Filter the first target encoder that supports the target encoding type and the first target decoder that supports the target decoding type through the MediaCodecUtils.codecSupportsType method in the custom WebRTC hardware video codec factory; Filter, by the MediaCodecUtils.selectColorFormat method, a second target encoder and a second target decoder that support the color format corresponding to the target data in the first target encoder and the first target decoder; A target encoder and a target decoder supporting hardware characteristics of the current electronic device are screened in the second target encoder and the second target decoder based on a user-defined function.

5. The method according to claim 4, characterized in that The method further comprises: The target encoder and target decoder are dynamically configured based on the hardware characteristics of the current electronic device through the createEncoder method.

6. The method according to claim 5, characterized in that The encoding process of the target data based on the target encoder to obtain the encoded target data includes: Extract frame data from the target data through VideoFrame and VideoFrame.Buffer; The extracted frame data is input into the target encoder for encoding processing to obtain encoded target data.

7. The method according to claim 5, characterized in that The method further comprises: Based on the SDK version of the current electronic device, support for a specific video encoding type is dynamically enabled when configuring the target encoder, wherein the specific video encoding type includes at least VP8 and VP9 types.

8. A data transmission device based on WebRTC, characterized in that: include: An acquisition module, used to acquire target data to be transmitted and hardware characteristics of the current electronic device; A determination module, configured to determine a target encoding type and a target decoding type based on the target data; A screening module, used for screening a target encoder and a target decoder in a custom WebRTC hardware video codec factory based on the target encoding type, the target decoding type and the hardware characteristics of the current electronic device; An encoding module, used for encoding the target data based on the target encoder to obtain encoded target data; The transmission module is used to transmit the encoded target data to a data receiving end, so that the data receiving end decodes and displays the received target data based on the target decoder.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a WebRTC-based data transmission program stored in the memory to implement the WebRTC-based data transmission method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the WebRTC-based data transmission method described in any one of claims 1 to 7.

Images