Media data transmission method and device, storage medium and electronic equipment

Abstract

The invention provides a media data transmission method and device, a storage medium and electronic equipment. The method comprises the following steps: in response to a communication instruction received in a first client running in a Linux system, sending a communication request to a second client, the communication request being used for establishing a communication connection between the first client and the second client; determining a target data interface according to a target communication type indicated by the communication instruction under the condition that a communication connection is established between the first client and the second client; target media data received from the first client side are obtained through the target data interface, the target media data are coded, target coded data are obtained, and the data type of the target media data is matched with the target communication type; sending the target coded data to a second client; the method can be applied to scenes such as automatic driving and smart traffic. According to the method and the device, the technical problem that the real-time media data cannot be transmitted in real time in the Linux system is solved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method and device for transmitting media data, a storage medium and an electronic device. Background Art

[0002] The software ecosystem in the Windows operating system platform or the Android operating system platform is very rich, and users can easily use the communication software in the above platforms to make voice calls or video calls.

[0003] With the development of Internet of Vehicles technology, the vehicle operating system can also provide a variety of applications to provide users with various services on the vehicle terminal. At present, there is a car system built on the Linux system in the vehicle operating system, but there is no relevant application in the Linux system to provide users with voice call or video call services. In other words, the relevant technical field cannot realize the real-time transmission of real-time video media data or voice media data in the Linux system.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for transmitting media data, a storage medium and an electronic device, so as to at least solve the technical problem that real-time media data cannot be transmitted in real time in a Linux system.

[0006] According to one aspect of an embodiment of the present invention, a method for transmitting media data is provided, comprising: in response to a communication instruction received in a first client running on a Linux system, sending a communication request to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client; in a case where the communication connection is established between the first client and the second client, determining a target data interface according to a target communication type indicated by the communication instruction; acquiring target media data received from the first client through the target data interface, and encoding the target media data to obtain target encoded data, wherein a data type of the target media data matches the target communication type; and sending the target encoded data to the second client.

[0007] According to one aspect of an embodiment of the present invention, a method for acquiring media data is provided, comprising: in response to receiving a communication request sent by a first client running on a Linux system, determining that a second client establishes a communication connection with the first client; receiving target coded data sent by the first client; decoding the target coded data according to a target communication type corresponding to the communication request to obtain target media data; and playing the target media data.

[0008] According to another aspect of an embodiment of the present invention, a media data transmission device is also provided, including: a first sending unit, used to send a communication request to a second client in response to a communication instruction received in a first client running on a Linux system, wherein the communication request is used to establish a communication connection between the first client and the second client; a determination unit, used to determine a target data interface according to a target communication type indicated by the communication instruction when the first client and the second client establish the communication connection; an encoding unit, used to obtain target media data received from the first client through the target data interface, and encode the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type; and a second sending unit, used to send the target encoded data to the second client.

[0009] Optionally, the above-mentioned determination unit includes: a first determination module, used to determine the audio data monitoring interface as the above-mentioned target data interface when the above-mentioned target communication type indicated by the above-mentioned communication instruction is a voice communication type; a second determination module, used to determine the video data acquisition interface as the above-mentioned target data interface when the above-mentioned target communication type indicated by the above-mentioned communication instruction is a video communication type.

[0010] Optionally, the above-mentioned first determination module is used to: create a target audio service, wherein the above-mentioned target audio service is used to provide multiple data interfaces, and each of the above-mentioned data interfaces is used to perform corresponding data operations; determine the above-mentioned audio data monitoring interface among the multiple data interfaces as the above-mentioned target data interface, wherein the above-mentioned audio data monitoring interface calls the audio monitoring module in the above-mentioned target audio service, and the above-mentioned audio monitoring module is used to collect audio data in the above-mentioned Linux system.

[0011] Optionally, the above-mentioned first determination module is used to: register a target callback function; when the above-mentioned audio data received in the above-mentioned first client is detected, call the above-mentioned audio data monitoring interface through the above-mentioned target callback function to obtain the above-mentioned audio data in the above-mentioned Linux system collected by the above-mentioned audio monitoring module.

[0012] Optionally, the second determination module is used to: obtain a function interface set provided by a video device driver in the Linux system; and determine a video data acquisition function in the function interface set as the target data interface.

[0013] Optionally, the second determination module is used to: open the virtual video device in the Linux system through the file reading function in the function interface set, wherein the virtual video device is used to obtain the video data received from the first client; obtain the device parameter set of the virtual video device through the parameter reading function in the function interface set, wherein the device parameter set is used to indicate the video data acquisition capability of the virtual video device; configure the target device parameters for the virtual video device through the parameter configuration function in the function interface set according to the device parameter set; obtain the video data collected by the target virtual video device through the video data acquisition function, wherein the target virtual video device is the virtual video device configured according to the target device parameters.

[0014] Optionally, the encoding unit includes: a first encoding module, used to create a virtual audio encoder when the target media data is target audio data; encoding the target media data through the virtual audio encoder to obtain the target encoded data; a second encoding module, used to traverse each video frame in the target video data when the target media data is target video data, convert the pixel information of the video frame from the first color space to the second color space to obtain reference video data; encoding the reference video data to obtain the target encoded data.

[0015] Optionally, the first encoding module is also used for at least one of the following: dynamically adjusting the volume value corresponding to each frame of audio data in the target audio data; performing noise suppression processing on the noise data in the target audio data; and suppressing the echo segment when an echo segment is detected in the target audio data, wherein the echo segment is an audio segment that is repeated in the audio data.

[0016] Optionally, the second encoding module is also used for at least one of the following: traversing each of the above-mentioned video frames in the above-mentioned target video data, filtering the pixel matrix corresponding to the above-mentioned video frame to obtain the target video frame, wherein the above-mentioned pixel matrix includes pixel values ​​corresponding to each pixel point in the above-mentioned video frame, and a first data amount of high-frequency noise data included in the target video frame is higher than a second data amount of high-frequency noise data included in the above-mentioned video frame before processing; obtaining a plurality of continuous video frames from the above-mentioned target video data; according to the comparison results of the pixel values ​​of the plurality of continuous video frames, filtering the pixel matrix corresponding to each of the plurality of continuous video frames to obtain a plurality of continuous target video frames, wherein a third data amount of low-frequency noise data included in the plurality of continuous target video frames is higher than a fourth data amount of low-frequency noise data included in the plurality of continuous video frames before processing.

[0017] According to another aspect of an embodiment of the present invention, a device for acquiring media data is also provided, including: a determination unit, used to determine that a second client establishes a communication connection with the first client in response to receiving a communication request sent by a first client running on a Linux system; a receiving unit, used to receive target encoded data sent by the first client; a decoding unit, used to decode the target encoded data according to a target communication type corresponding to the communication request to obtain target media data; and a playing unit, used to play the target media data.

[0018] Optionally, the above-mentioned decoding unit includes: a first decoding module, used to create a virtual audio decoding object when the above-mentioned target communication type is a voice communication type; configure decoding parameters for the above-mentioned virtual audio decoding object; and decode the above-mentioned target encoded data through the above-mentioned virtual audio decoding object according to the decoding method indicated by the above-mentioned decoding parameters to obtain target audio data.

[0019] Optionally, the above-mentioned decoding unit includes: a second decoding module, which is used to decompress the above-mentioned target encoded data to obtain reference encoded data when the above-mentioned target communication type is a video communication type; determine the encoding type of the above-mentioned reference encoded data, and decode the above-mentioned reference encoded data according to a video decoder matching the above-mentioned encoding type to obtain target video data.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned media data transmission method or the above-mentioned media data acquisition method when running.

[0021] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned method for transmitting media data or the above-mentioned method for acquiring media data.

[0022] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the media data transmission method or the media data acquisition method through the computer program.

[0023] In an embodiment of the present invention, in response to a communication instruction received in a first client running on a Linux system, a communication request is sent to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client; when the first client and the second client establish the communication connection, a target data interface is determined according to a target communication type indicated by the communication instruction; target media data received from the first client is acquired through the target data interface, and the target media data is encoded to obtain target encoded data, wherein the data type of the target media data matches the target communication type; and the target encoded data is sent to the second client, thereby realizing real-time transmission of media data in the Linux system.

[0024] In the above implementation, when a communication connection is established between a first client and a second client running on a Linux system, a corresponding target data interface can be called according to the communication type corresponding to the communication connection, thereby obtaining target media data received from the first client according to the target media interface, and based on the encoding and transmission of the target media data, the transmission of real-time media data is realized under the Linux system, thereby solving the technical problem of real-time transmission of real-time received media data under the Linux system in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a schematic diagram of a hardware environment of an optional method for transmitting media data according to an embodiment of the present invention;

[0027] Figure 2 is a flow chart of an optional method for transmitting media data according to an embodiment of the present invention;

[0028] Figure 3 is a flowchart of an optional method for acquiring video data according to an embodiment of the present invention;

[0029] Figure 4 is a flowchart of an optional method for acquiring media data according to an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of an optional method for transmitting media data according to an embodiment of the present invention;

[0031] Figure 6 A schematic structural diagram of an optional media data transmission device according to an embodiment of the present invention;

[0032] Figure 7 is a schematic structural diagram of an optional electronic device according to an embodiment of the present invention;

[0033] Figure 8 A schematic structural diagram of an optional device for receiving media data according to an embodiment of the present invention;

[0034] Fig. 9 is a schematic structural diagram of another optional electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] For ease of understanding, the nouns that will appear in the embodiments of this application are explained as follows:

[0038] Intelligent Traffic System (ITS): also known as Intelligent Transportation System, is the effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thus forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0039] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS): referred to as vehicle infrastructure cooperative systems, is a development direction of intelligent transportation systems (ITS). The vehicle infrastructure cooperative system adopts advanced wireless communication and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and carries out active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0040] According to one aspect of an embodiment of the present application, a method for transmitting media data is provided. As an optional implementation, the above-mentioned method for transmitting media data can be applied to, but is not limited to, Figure 1 The data transmission hardware system shown in FIG. 1 is composed of a terminal device 102, a network 104, a server 106, a database 108, and a terminal device 110. Figure 1 As shown, the terminal device 102 and the terminal device 110 can be connected via a network 104, and the server 112 is connected to the terminal device 102 and the terminal device 110 via the network 104. The above-mentioned network 104 may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that realize wireless communication.

[0041] The terminal device 102 and the terminal device 110 may include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft and other types of terminal devices. Specifically, the operating system running in the terminal device corresponding to the terminal device 102 is specifically a Linux system, and a first client for real-time communication runs in the Linux system; a second client for real-time communication also runs in the terminal device 110, and the operating system running in the terminal device 110 may include but is not limited to one of the Linux system, Windows system, Android system, IOS system and the like.

[0042] The terminal device 102 and the terminal device 110 are also provided with a display, a processor and a memory. The display can be used to display the interactive interface of the first client or the second client, the processor can be used to acquire and encode the collected media data; the memory is used to temporarily store the acquired media data.

[0043] In a specific implementation, step S102 may be executed in the first client running on the terminal device 102 to receive a communication instruction, wherein the communication request is used to establish a communication connection between the first client and the second client; then, step S104 is executed to send a communication request to the server 106 through the network 104 in response to the communication instruction received in the first client running on the Linux system, and the server 106 may be used to forward the communication request to the terminal device 110; then, the terminal device 110 executes S106 to return a connection confirmation instruction to the server 106 through the network 104, and the server 106 returns the connection confirmation instruction to the terminal device 102 through the network 104;

[0044] Next, steps S108 to S112 are executed in the terminal device 102. When a communication connection is established between the first client and the second client, the target data interface is determined according to the target communication type indicated by the communication instruction; the target media data received from the first client is obtained through the target data interface, and the target media data is encoded to obtain target encoded data, wherein the data type of the target media data matches the target communication type; and the target encoded data is sent to the second client.

[0045] The server 112 may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0046] In an embodiment of the present invention, in response to a communication instruction received in a first client running on a Linux system, a communication request is sent to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client; when the first client and the second client establish the communication connection, a target data interface is determined according to a target communication type indicated by the communication instruction; target media data received from the first client is acquired through the target data interface, and the target media data is encoded to obtain target encoded data, wherein the data type of the target media data matches the target communication type; and the target encoded data is sent to the second client, thereby realizing real-time transmission of media data in the Linux system.

[0047] In the above implementation, when a communication connection is established between a first client and a second client running on a Linux system, a corresponding target data interface can be called according to the communication type corresponding to the communication connection, thereby obtaining target media data received from the first client according to the target media interface, and based on the encoding and transmission of the target media data, the transmission of real-time media data is realized under the Linux system, thereby solving the technical problem of real-time transmission of real-time received media data under the Linux system in the prior art.

[0048] As an optional implementation, Figure 2 As shown, the above-mentioned method for transmitting media data can be applied to a terminal device running a Linux system, and specifically may include the following steps:

[0049] S202, in response to a communication instruction received in the first client running on the Linux system, sending a communication request to the second client, wherein the communication request is used to establish a communication connection between the first client and the second client;

[0050] S204, when the first client and the second client establish a communication connection, determine a target data interface according to a target communication type indicated by the communication instruction;

[0051] S206, acquiring target media data received from the first client through the target data interface, and encoding the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type;

[0052] S208: Send the target encoded data to the second client.

[0053] It should be noted that the above implementation of the present application can be applied to the scenario where the user object communicates in real time through two clients, and the application types corresponding to the above clients can include but are not limited to instant messaging applications, game applications that can provide real-time conversation functions, live broadcast applications that can provide real-time conversation functions, etc. The above first client is an application client running on a Linux system, specifically, the above Linux system can be a debian system, an archlinux system, an ubuntu system, or one of the commercially developed Red Hat systems, SUSE systems, and Oracle Linux systems; the above second client is running on one of the systems including but not limited to a Linux system, a Windows system, an Android system, or an IOS system.

[0054] The terminal devices running the above Linux system may include but are not limited to at least one of the following: mobile terminals (such as mobile phones), laptops, tablet computers, PDAs, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, vehicle-mounted terminals, aircraft terminals, etc.

[0055] The following describes implementation scenarios to which the above implementation methods can be applied in conjunction with specific scenarios.

[0056] In the case where the first terminal is a vehicle-mounted terminal and the second terminal is a mobile phone terminal, a vehicle-mounted system based on a Linux system may be run in the vehicle-mounted terminal, and a first navigation client may be run in the vehicle-mounted system, and the first navigation client may provide a real-time conversation function. In response to the communication instruction received in the first navigation client, a communication request is initiated to the second navigation client running in the mobile phone terminal, and a communication connection is established between the first navigation client and the second navigation client. Furthermore, in response to the first voice message "Hello, I am almost at the place you specified" received from the first navigation client, the first voice message is obtained and encoded, and sent to the second navigation client;

[0057] In the case where the first terminal is a personal computer terminal and the second terminal is a personal computer terminal, a Linux system may be running in the first terminal, and a first game client may be running in the Linux system, and the first game client may provide a real-time conversation function between teammates. In response to the communication instruction received in the first game client, a communication request is initiated to the second game client running in the second terminal, and a communication connection is established between the first game client and the second game client. Furthermore, in response to the second voice message "There are supplies in City X, come and get them" received from the first game client, the second voice message is obtained and encoded, and sent to the second game client;

[0058] In the case where the first terminal is a personal computer terminal and the second terminal is a mobile phone terminal, a Linux system may be run in the first terminal, and a viewer client may be run in the Linux system, and the viewer client may provide a real-time video interaction function between the viewer account and the anchor account. In response to the communication instruction received in the viewer client, a communication request is initiated to the anchor client running in the second terminal, and a communication connection is established between the viewer client and the anchor client. Furthermore, in response to the video stream received from the viewer client, the video stream is obtained and encoded, and sent to the anchor client.

[0059] Through the above-mentioned implementation mode of the present application, when a communication connection is established between a first client and a second client running on a Linux system, a corresponding target data interface can be called according to the communication type corresponding to the communication connection, thereby obtaining target media data received from the first client according to the target media interface, and based on the encoding and transmission of the target media data, the transmission of real-time media data under the Linux system is realized, thereby solving the technical problem of real-time transmission of real-time received media data under the Linux system in the prior art.

[0060] In an optional implementation, in the case where the first client and the second client establish a communication connection, determining the target data interface according to the target communication type indicated by the communication instruction includes:

[0061] Case 1: When the target communication type indicated by the communication instruction is a voice communication type, the audio data monitoring interface is determined as the target data interface;

[0062] Case 2: When the target communication type indicated by the communication instruction is a video communication type, the video data acquisition interface is determined as the target data interface.

[0063] It is understandable that in response to different communication types of real-time communication triggered by the user, the corresponding data interface can be called to obtain real-time media data. In the case where the real-time communication is real-time voice communication, the audio data monitoring interface can be obtained and determined as the target data interface; in the case where the real-time communication is real-time video communication, the video data acquisition interface can be obtained and determined as the target data interface.

[0064] Optionally, when the target communication type indicated by the communication instruction is a voice communication type, determining the audio data monitoring interface as the target data interface includes:

[0065] S1, creating a target audio service, wherein the target audio service is used to provide multiple data interfaces, each data interface is used to perform a corresponding data operation;

[0066] S2, determining an audio data monitoring interface among the multiple data interfaces as a target data interface, wherein the audio data monitoring interface calls an audio monitoring module in a target audio service, and the audio monitoring module is used to collect audio data in a Linux system.

[0067] In this embodiment, the audio data monitoring interface can be specifically a functional interface provided by the Pulse system. It should be noted that the Pulse system is an audio acquisition and playback system for the Linux operating system, which uses PulseAudio as an audio server and implements audio acquisition and playback through the API provided by PulseAudio.

[0068] Specifically, in the above implementation, the target audio service created by the first client can be specifically a PulseAudio service, and then the tool interface provided by the Pulse system can be called through the created PulseAudio service. It should be noted that the Pulse system can further include an audio monitoring module (monitor module), which can be used to capture any audio being played in the Linux system, including system sounds, music players, network streaming media, etc. The module function of the audio monitoring module can be called through the tool interface.

[0069] In an optional implementation manner, the step of acquiring the target media data received from the first client through the target data interface includes:

[0070] S1, register the target callback function;

[0071] S2, when the audio data received in the first client is detected, the audio data monitoring interface is called through the target callback function to obtain the audio data in the Linux system collected by the audio monitoring module.

[0072] It can be understood that, in this embodiment, the specific process of obtaining the audio currently input by the user object through the above-mentioned Pulse service can be: first create a PulseAudio service, then use the API provided by PulseAudio to connect to the monitor module, and register a callback function to obtain the collected audio data.

[0073] It should be noted that, in this implementation, Pulse audio capture can be used to capture any audio being played, including system sounds, music players, network streaming media, etc. Pulse audio capture can capture audio data through the monitor module provided by PulseAudio, and then send it to the application for processing. The advantage of this solution is that it can capture system-level audio data while having good compatibility and stability.

[0074] In a specific application scenario, when the communication between the first client and the second client is voice communication, the audio source transmitted to the second client can be determined according to the selection operation of the user account. That is, in this embodiment, the user account can choose to send the call voice, or send the call voice and system sound, or send the call voice and the sound of the music player during the call, thereby realizing real-time sharing of various types of sounds during the call.

[0075] Through the above implementation of the present application, by calling the Pulse service during the operation of the above-mentioned first client, real-time acquisition of the current voice data of the user object is achieved, thereby providing good compatibility and stability for the application during the voice call.

[0076] In an optional implementation, when the target communication type indicated by the communication instruction is a video communication type, determining the video data acquisition interface as the target data interface includes:

[0077] S1, obtaining a function interface set provided by a video device driver in a Linux system;

[0078] S2, determining the video data acquisition function in the function interface set as the target data interface.

[0079] It should be noted that, when the target communication type is a video communication type, the acquisition of video data can be achieved by calling a function interface provided by a kernel driver of the Linux system. The kernel driver can be specifically a V4L2 driver (Video for Linux 2).

[0080] The above V4L2 driver can specifically provide the following interface sets:

[0081] The first interface set: video capture interface, which can obtain the collected data of the device, and the above device can be a tuner or a camera;

[0082] The second interface set: video output interface, which can drive the computer's peripheral video image devices - such as devices that can output TV signal formats;

[0083] The third interface set: direct video transmission interface (video overlay interface), its main function is to output the signal collected from the video acquisition device directly to the output device without passing through the system CPU;

[0084] The fourth interface set: Video Interval Blanking Signal Interface (VBI interface), which allows applications to access the video signal during the transmission blanking period;

[0085] The fifth interface set: radio interface, which can be used to process the audio stream received from the AM or FM tuner device.

[0086] In this implementation, the target data interface may be specifically acquired from the first interface set.

[0087] In an optional implementation manner, after obtaining the function interface set provided by the video device driver in the Linux system, the method further includes:

[0088] S1, opening a virtual video device in the Linux system through a file reading function in a function interface concentration, wherein the virtual video device is used to obtain video data received from the first client;

[0089] S2, obtaining a device parameter set of the virtual video device through a parameter reading function in a function interface set, wherein the device parameter set is used to indicate a video data acquisition capability of the virtual video device;

[0090] S3, configuring target device parameters for the virtual video device through a parameter configuration function in a function interface set according to the device parameter set;

[0091] S4, obtaining the video data collected by the target virtual video device through a video data collection function, wherein the target virtual video device is a virtual video device configured according to target device parameters.

[0092] The following combination Figure 3 The process of obtaining video images through the function interface set provided by the V4L2 driver in the Linux system is described.

[0093] S302, turn on the video device;

[0094] For example, you can use the open() function (i.e. the file reading function mentioned above) to open a video device file, such as / dev / video0;

[0095] S304, querying device parameters;

[0096] Use the parameter query function in the ioctl() function (i.e. the parameter reading function mentioned above) to query the device parameters, such as the video format, resolution, frame rate, etc. supported by the virtual video device in the above Linux system. It can be understood that the parameter reading function mentioned above can be used to determine what functions the current virtual video device has, such as whether it has video input, or audio input and output, etc.;

[0097] S306, setting device parameters;

[0098] Use the parameter configuration function to set the device parameters of the video device. For example, select the video format, resolution, frame rate, etc. of the video data collected during the current video session according to the device parameters obtained by the parameter query function.

[0099] S308, start collecting;

[0100] It should be noted that before video capture, you can first apply for a frame buffer from the driver, usually no more than 5 frames; then, map the applied frame buffer to the user space; finally, put all the applied frame buffers into the queue to store the captured data;

[0101] Next, use the ioctl() function (i.e. the above-mentioned video data acquisition function) to start video acquisition and begin reading data from the video device.

[0102] In addition, you can also get the frame buffer of the collected data by dequeuing, and get the original collected data; and put the buffer back into the end of the queue, so that you can collect in a loop;

[0103] S310, reading collected data;

[0104] In this implementation, the read() function may be used to read the collected data from the video device and process it.

[0105] S312, stop collecting;

[0106] In this step, you can use the stop acquisition function to stop video acquisition and stop reading data from the video device;

[0107] S314, close the device; the close() function can be used to close the video device file.

[0108] In an optional implementation manner, the encoding process of the target media data to obtain the target encoded data includes:

[0109] Case 1: when the target media data is target audio data, a virtual audio encoder is created; the target encoded data is obtained by encoding the target audio data through the virtual audio encoder;

[0110] Case 2: When the target media data is target video data, each video frame in the target video data is traversed, and pixel information of the video frame is converted from the first color space to the second color space to obtain reference video data; the reference video data is encoded to obtain target encoded data.

[0111] In this implementation, corresponding to different communication types, different encoding methods may be used to encode the media data.

[0112] In the case where the target media data is target audio data, Opus may be specifically used to encode the audio at the transmitting end. It should be noted that Opus is an audio encoding format, which is standardized by IETF (Internet Engineering Task Force).

[0113] In this embodiment, during the process of encoding audio data, an Opus encoder object may be created first, and the encoder may be initialized. For example, an Opus encoder with a sampling rate of 48000 Hz and dual channels may be created; then, according to the needs of the session, the encoder parameters such as bit rate, complexity, etc. may be configured; and then the collected PCM audio (Pulse Code Modulation) data may be Opus encoded.

[0114] When the target media data is target video data, the encoding of the video data can be achieved through two processes: preprocessing and compression encoding.

[0115] In the preprocessing process, color space conversion can be performed first, that is, each video frame in the video data is converted from RGB color space to YUV color space. It should be noted that YUV color space is more suitable for video encoding and can improve encoding efficiency.

[0116] It should be noted that the above RGB color space means that each pixel in the above video frame can be encoded based on the three basic colors of R (Red), G (Green), and B (Blue); "Y" in the above YUV color space represents brightness (Luminance or Luma), that is, the grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image. In other words, each pixel in the above video frame can be encoded by the above "Y", "U" and "V";

[0117] After the pixel points are converted into color space, downsampling is then performed to reduce the resolution of the video data and improve transmission efficiency.

[0118] Next, in the compression coding stage, the steps may include motion estimation, block segmentation, discrete cosine transform, quantization, bitrate control, etc. Specifically, the coding formats of H264 and H265 standards may be supported, wherein H265 is an upgraded version of H264 with a higher compression rate, but with a larger amount of calculation. In practical applications, the coding format may be selected according to the usage scenario.

[0119] In an optional implementation manner, when the target media data is target audio data, before creating the virtual audio encoder, the method further includes at least one of the following:

[0120] Method 1: dynamically adjust the volume value corresponding to each frame of audio data in the target audio data;

[0121] Method 2: performing noise suppression processing on the noise data in the target audio data;

[0122] Method three: when an echo segment is detected in the target audio data, suppressing the echo segment, wherein the echo segment is an audio segment that is repeated in the audio data.

[0123] It should be noted that the above method 1 can be specifically an automatic gain control (ACG) processing method, which can be used to adjust the volume of the input signal to maintain a consistent volume level and avoid the sound being too strong or too weak. Specifically, the input signal can be adjusted by using an effective combination of linear amplification and compression amplification. When a weak signal is input, the linear amplification circuit works to ensure the strength of the output signal; when the input signal reaches a certain strength, the compression amplification circuit is started to reduce the output amplitude;

[0124] The second method can be specifically a noise suppression control (Automatic Noise Suppression, ANS) method, which can be used to eliminate background noise. Specifically, the method can detect the background noise of fixed frequency and eliminate the background noise, automatically filter out the fixed frequency fan and air conditioner sound, and present clear voice between the call parties.

[0125] The above method 3 can be specifically an automatic echo cancellation (Acoustic Echo Cancel, AEC) method, which can be used to eliminate echoes during a call. Specifically, the audio returned by the second client can be filtered by an adaptive filter, and the echo signal contained in the returned audio can be removed.

[0126] It can be understood that, in a preferred manner, the above three methods can be combined to process the audio data before encoding the audio data, thereby improving the quality and clarity of the voice call.

[0127] In an optional implementation manner, before converting the pixel information of the video frame from the first color space to the second color space to obtain the reference video data, the process further includes at least one of the following:

[0128] Method 1: traverse each video frame in the target video data, and perform filtering processing on a pixel matrix corresponding to the video frame to obtain a target video frame, wherein the pixel matrix includes pixel values ​​corresponding to each pixel point in the video frame, and a first data amount of high-frequency noise data included in the target video frame is higher than a second data amount of high-frequency noise data included in the video frame before processing;

[0129] Method 2: obtaining multiple continuous video frames from target video data; and filtering pixel matrices corresponding to the multiple continuous video frames based on comparison results of pixel values ​​of the multiple continuous video frames to obtain multiple continuous target video frames, wherein a third data amount of low-frequency noise data included in the multiple continuous target video frames is higher than a fourth data amount of low-frequency noise data included in the multiple continuous video frames before processing.

[0130] It can be understood that, in this embodiment, before encoding the video data, a 3DNR method can be used.

[0131] Specifically, in this embodiment, the video frame can be processed by combining filters based on space and time. In space, the local pixel information of the image is used to determine the location and intensity of the noise, and then the noise is filtered. In time, the time series information of the video is used to determine the change trend of the noise, and then it is filtered. This method of using spatial and temporal information in combination can more accurately identify and eliminate noise in the video.

[0132] Specifically, the following steps may be included:

[0133] S1, processing by spatial domain filtering: in each frame of the image, the pixel values ​​are filtered to eliminate the high-frequency noise in the image;

[0134] S2, processing by time domain filtering: filtering the pixel values ​​in multiple frames to eliminate low-frequency noise in the image;

[0135] S3, dynamic noise estimation: by analyzing the noise in the image, determine the location and intensity of the noise.

[0136] S4, according to the location and intensity of the noise, different filtering methods are used to eliminate the noise.

[0137] In an optional implementation manner, the sending of the target coded data to the second client includes:

[0138] S1, performing packet processing on the target coded data to obtain multiple coded data packets;

[0139] S2, sending multiple encoded data packets to the second client.

[0140] It is understandable that in this embodiment, the encoded audio and video data can be packaged, the voice or video data can be divided into data packets of a certain size, and transmitted to the receiving party through the network, and the receiving party then assembles the data packets into a complete voice or video signal. In audio and video calls, package transmission is a very important link, which directly affects the call quality and stability.

[0141] In packet transmission, the following aspects can be considered:

[0142] The first aspect is packet size. The smaller the packet size, the better. This can reduce network delay and packet loss probability. In a preferred embodiment, the voice packet size is 20-30ms and the video packet size is 40-60ms.

[0143] The second aspect: packet header information, the packet header contains some necessary information, such as the packet sequence number, timestamp, data type, etc. The size of the packet header should be as small as possible to reduce the impact of the packet header on data transmission.

[0144] The third aspect: network transmission protocol. Audio and video data need to be transmitted through the network. The stability and delay of network transmission will affect the quality of the call. In order to reduce network delay, some network optimization technologies can be used, such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP).

[0145] Fourth aspect: packet loss retransmission processing. Due to the instability of network transmission, data packets may be lost. In order to ensure the quality of the call, the lost data packets need to be retransmitted. The timing and method of retransmission need to be adjusted according to the specific situation.

[0146] In summary, the transmission of audio and video call group packets needs to comprehensively consider factors such as packet size, packet header, network transmission, and packet loss retransmission to ensure the quality and stability of the call.

[0147] As an optional implementation, Figure 4 As shown, a method for obtaining media data is provided, which may specifically include the following steps:

[0148] S402, in response to receiving a communication request sent by a first client running on a Linux system, determining that a second client establishes a communication connection with the first client;

[0149] S404, receiving target coded data sent by the first client;

[0150] S406, decoding the target coded data according to the target communication type corresponding to the communication request to obtain target media data;

[0151] S408: Play the target media data.

[0152] It is understandable that the above-mentioned second client runs on one of the systems including but not limited to Linux system, Windows system, Android system or IOS system.

[0153] In an optional implementation manner, the above decoding the target coded data according to the target communication type corresponding to the communication request to obtain the target media data includes:

[0154] S1, when the target communication type is a voice communication type, creating a virtual audio decoding object;

[0155] S2, configuring decoding parameters for the virtual audio decoding object;

[0156] S3, decoding the target encoded data according to the decoding method indicated by the decoding parameters through the virtual audio decoding object to obtain the target audio data.

[0157] In this embodiment, in the above-mentioned second client, the received audio data can be unpacked, and then the Opus library can be used to create a decoder object, and the parameters of the decoder, such as sampling rate, number of channels, etc., can be set; then the Opus audio data can be decoded through the decoder object to obtain audio data in PCM format; and the audio data in PCM format can be handed over to PulseAudio for playback.

[0158] In an optional implementation manner, the above decoding the target coded data according to the target communication type corresponding to the communication request to obtain the target media data includes:

[0159] S1, when the target communication type is a video communication type, decompressing the target coded data to obtain reference coded data;

[0160] S2, determining the coding type of the reference coded data, and decoding the reference coded data according to a video decoder matching the coding type to obtain target video data.

[0161] In this embodiment, in the above-mentioned second client, the received video data can be unpacked, and then the encoding type can be determined, and the corresponding decoder can be used to decode the video; then the video data can be transmitted to the Web interface through RPC (Remote Procedure Call); finally, WebGL can be used to render the video data.

[0162] The following combination Figure 5 The media data transmission system provided by this application is described.

[0163] like Figure 5 As shown, in this embodiment, the process of audio communication mainly includes three steps: Pulse audio acquisition, audio 3A processing and Opus audio encoding.

[0164] In the process of Pulse audio acquisition, the specific process of obtaining the audio currently input by the user object through the above-mentioned Pulse service can be: first create a PulseAudio service, then use the API provided by PulseAudio to connect the monitor module, and register the callback function to obtain the collected audio data. It should be noted that in this embodiment, Pulse audio acquisition can be used to capture any audio being played, including system sounds, music players, network streaming media, etc. Pulse audio acquisition can capture audio data through the monitor module provided by PulseAudio, and then send it to the application for processing.

[0165] In the audio 3A processing process, 3A processing can be achieved by combining the following three methods. Specifically including: Automatic Gain Control (ACG) processing, noise suppression control (ANS), and automatic echo cancellation (AEC). Among them, automatic gain control can be used to adjust the volume of the input signal to maintain a consistent volume level and avoid the sound being too strong or too weak; noise suppression control is used to eliminate background noise. Specifically, the above method can detect the background noise of fixed frequency and eliminate the background noise, automatically filter out the fixed frequency fan and air conditioning sounds, and present a clear sound between the call objects; automatic echo cancellation can be used to eliminate the echo during the call. Specifically, the audio returned by the second client can be filtered by an adaptive filter, and then the echo signal contained in the returned audio can be cleared. By combining the above three methods to process the audio data, the quality and clarity of the voice call can be improved.

[0166] In the Opus audio encoding process, you can first create an Opus encoder object and initialize the encoder. For example, you can create an Opus encoder with a sampling rate of 48000Hz and dual channels; then configure the encoder parameters such as bit rate, complexity, etc. according to the session needs; then Opus encode the collected PCM audio (Pulse Code Modulation) data.

[0167] The process of video communication mainly includes three steps: V4L2 video acquisition, video pre-processing: denoising and video encoding.

[0168] During the V4L2 video acquisition process, if Figure 3As shown, the following steps may be specifically included: S302, opening the video device; for example, the open() function (i.e., the file reading function) may be used to open the video device file, such as / dev / video0; S304, querying the device parameters; using the parameter query function in the ioctl() function (i.e., the parameter reading function) to query the device parameters, such as the video format, resolution, frame rate, etc. supported by the virtual video device in the above Linux system. It can be understood that the parameter reading function may be used to determine what functions the current virtual video device has, such as whether it has video input, or audio input and output, etc.; S306, setting the device parameters; using the parameter configuration function to set the device parameters of the video device, such as selecting the video format, resolution, frame rate, etc. of the video data collected during the current video session according to the device parameters obtained by the parameter query function. S308, start acquisition; it should be noted that before video acquisition, you can first apply for a frame buffer from the driver, generally not more than 5 frames; then, map the applied frame buffer to the user space; finally, put all the applied frame buffers into the queue to store the acquired data; then, use the ioctl() function (i.e. the above-mentioned video data acquisition function) to start video acquisition and start reading data from the video device. In addition, you can also get the frame buffer of the acquired data by dequeuing to get the original acquired data; and put the buffer back into the end of the queue, so that you can collect in a loop; S310, read the acquired data; in this embodiment, you can use the read() function to read the acquired data from the video device and process it. S312, stop acquisition; in this step, you can use the stop acquisition function to stop video acquisition and stop reading data from the video device; S314, close the device; you can use the close() function to close the video device file.

[0169] In the process of video pre-processing, the video frames can be processed by combining filters based on space and time. In space, the local pixel information of the image is used to determine the location and intensity of the noise, and then the noise is filtered. In time, the time series information of the video is used to determine the change trend of the noise, and then it is filtered. This method of combining spatial and temporal information can more accurately identify and eliminate noise in the video.

[0170] In the video encoding process, the encoding of video data can be achieved through two processes: preprocessing and compression encoding. In the preprocessing process, color space conversion can be performed first, that is, each video frame in the video data is converted from RGB color space to YUV color space. It should be noted that YUV color space is more suitable for video encoding and can improve encoding efficiency. After the color space conversion of the pixel points, downsampling processing is then performed to reduce the resolution of the video data and improve transmission efficiency. Then, in the compression encoding stage, it can specifically include steps such as motion estimation, block segmentation, discrete cosine transform, quantization, and bit stream control.

[0171] like Figure 5 In the process of transmitting the encoded data through the network, the audio encoded data can be decoded and played. Specifically, the received audio data can be unpacked in the second client, and then the decoder object can be created using the Opus library, and the decoder parameters such as sampling rate and number of channels can be set; then the Opus audio data can be decoded through the decoder object to obtain audio data in PCM format; and the audio data in PCM format can be handed over to PulseAudio for playback.

[0172] For video encoding data, video decoding can be performed and video rendering can be performed through WebGL. Specifically, the received video data can be unpacked in the second client, and then the encoding type can be determined and the corresponding decoder can be used to decode the video; then the video data can be transmitted to the Web interface through rpc; finally, WebGL can be used to render the video data and play the video.

[0173] Through the above-mentioned implementation mode of the present application, real-time audio and video calls are realized on the Linux platform; in addition, by performing audio 3A processing on the collected audio data in the Linux platform, a certain volume level and audio quality are maintained under different audio sources and environments, thereby improving the audibility and intelligibility of the audio; further, by performing video noise reduction on the collected video data in the Linux platform, the video clarity can be improved and the bit rate can be reduced.

[0174] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0175] According to another aspect of the embodiments of the present invention, a media data transmission device for implementing the above-mentioned media data transmission method is also provided. Figure 6 As shown, the device comprises:

[0176] A first sending unit 602 is used to send a communication request to a second client in response to a communication instruction received in a first client running on a Linux system, wherein the communication request is used to establish a communication connection between the first client and the second client;

[0177] A determining unit 604 is configured to determine a target data interface according to a target communication type indicated by the communication instruction when the first client and the second client establish the communication connection;

[0178] The encoding unit 606 is used to obtain the target media data received from the first client through the target data interface, and encode the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type;

[0179] The second sending unit 608 is used to send the target coded data to the second client.

[0180] Optionally, the above-mentioned determination unit 604 includes: a first determination module, used to determine the audio data monitoring interface as the above-mentioned target data interface when the above-mentioned target communication type indicated by the above-mentioned communication instruction is a voice communication type; a second determination module, used to determine the video data acquisition interface as the above-mentioned target data interface when the above-mentioned target communication type indicated by the above-mentioned communication instruction is a video communication type.

[0181] Optionally, the above-mentioned first determination module is used to: create a target audio service, wherein the above-mentioned target audio service is used to provide multiple data interfaces, and each of the above-mentioned data interfaces is used to perform corresponding data operations; determine the above-mentioned audio data monitoring interface among the multiple data interfaces as the above-mentioned target data interface, wherein the above-mentioned audio data monitoring interface calls the audio monitoring module in the above-mentioned target audio service, and the above-mentioned audio monitoring module is used to collect audio data in the above-mentioned Linux system.

[0182] Optionally, the above-mentioned first determination module is used to: register a target callback function; when the above-mentioned audio data received in the above-mentioned first client is detected, call the above-mentioned audio data monitoring interface through the above-mentioned target callback function to obtain the above-mentioned audio data in the above-mentioned Linux system collected by the above-mentioned audio monitoring module.

[0183] Optionally, the second determination module is used to: obtain a function interface set provided by a video device driver in the Linux system; and determine a video data acquisition function in the function interface set as the target data interface.

[0184] Optionally, the second determination module is used to: open the virtual video device in the Linux system through the file reading function in the function interface set, wherein the virtual video device is used to obtain the video data received from the first client; obtain the device parameter set of the virtual video device through the parameter reading function in the function interface set, wherein the device parameter set is used to indicate the video data acquisition capability of the virtual video device; configure the target device parameters for the virtual video device through the parameter configuration function in the function interface set according to the device parameter set; obtain the video data collected by the target virtual video device through the video data acquisition function, wherein the target virtual video device is the virtual video device configured according to the target device parameters.

[0185] Optionally, the encoding unit 606 includes: a first encoding module, used to create a virtual audio encoder when the target media data is target audio data; encode the target media data through the virtual audio encoder to obtain the target encoded data; a second encoding module, used to traverse each video frame in the target video data when the target media data is target video data, convert the pixel information of the video frame from the first color space to the second color space to obtain reference video data; encode the reference video data to obtain the target encoded data.

[0186] Optionally, the first encoding module is also used for at least one of the following: dynamically adjusting the volume value corresponding to each frame of audio data in the target audio data; performing noise suppression processing on the noise data in the target audio data; and suppressing the echo segment when an echo segment is detected in the target audio data, wherein the echo segment is an audio segment that is repeated in the audio data.

[0187] Optionally, the second encoding module is also used for at least one of the following: traversing each of the above-mentioned video frames in the above-mentioned target video data, filtering the pixel matrix corresponding to the above-mentioned video frame to obtain the target video frame, wherein the above-mentioned pixel matrix includes pixel values ​​corresponding to each pixel point in the above-mentioned video frame, and a first data amount of high-frequency noise data included in the target video frame is higher than a second data amount of high-frequency noise data included in the above-mentioned video frame before processing; obtaining a plurality of continuous video frames from the above-mentioned target video data; according to the comparison results of the pixel values ​​of the plurality of continuous video frames, filtering the pixel matrix corresponding to each of the plurality of continuous video frames to obtain a plurality of continuous target video frames, wherein a third data amount of low-frequency noise data included in the plurality of continuous target video frames is higher than a fourth data amount of low-frequency noise data included in the plurality of continuous video frames before processing.

[0188] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0189] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned media data transmission method is also provided. The electronic device may be Figure 7 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Figure 7 As shown, the electronic device includes a memory 702 and a processor 704. The memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the above method embodiments through the computer program.

[0190] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0191] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0192] S1, in response to a communication instruction received in a first client running on a Linux system, sending a communication request to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client;

[0193] S2, when the first client and the second client establish the communication connection, determining a target data interface according to a target communication type indicated by the communication instruction;

[0194] S3, acquiring target media data received from the first client through the target data interface, and encoding the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type;

[0195] S4, sending the target coded data to the second client.

[0196] Alternatively, a person skilled in the art may understand that: Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 7 The electronic device and the electronic equipment described above are not limited in structure. Figure 7 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 7 Different configurations are shown.

[0197] Among them, the memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the media data transmission method and device in the embodiment of the present invention. The processor 704 executes various functional applications and the transmission of media data by running the software programs and modules stored in the memory 702, that is, realizing the above-mentioned media data transmission method. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include a memory remotely arranged relative to the processor 704, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 702 can be specifically, but not limited to, used to store information such as various elements in the scene screen, data acquisition information, etc. As an example, such as Figure 7 As shown, the memory 702 may include, but is not limited to, the first sending unit 602, the determining unit 604, the encoding unit 606, and the second sending unit 608 in the media data transmission device. In addition, other module units in the media data transmission device may also be included but are not limited to, which will not be repeated in this example.

[0198] Optionally, the transmission device 706 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 706 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0199] In addition, the electronic device further includes: a display 708 for displaying a virtual scene in an interface; and a connection bus 710 for connecting various module components in the electronic device.

[0200] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, a peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0201] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0202] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0203] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for transmitting media data.

[0204] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0205] S1, in response to a communication instruction received in a first client running on a Linux system, sending a communication request to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client;

[0206] S2, when the first client and the second client establish the communication connection, determining a target data interface according to a target communication type indicated by the communication instruction;

[0207] S3, acquiring target media data received from the first client through the target data interface, and encoding the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type;

[0208] S4, sending the target coded data to the second client.

[0209] According to another aspect of the embodiments of the present invention, a device for receiving media data for implementing the above-mentioned method for transmitting media data is also provided. Figure 8 As shown, the device comprises:

[0210] The determining unit 802 is configured to determine, in response to receiving a communication request sent by a first client running on a Linux system, that a second client establishes a communication connection with the first client;

[0211] The receiving unit 804 is used to receive the target coded data sent by the first client;

[0212] A decoding unit 806, configured to decode the target coded data according to the target communication type corresponding to the communication request to obtain target media data;

[0213] The playing unit 806 is used to play the target media data.

[0214] Optionally, the above-mentioned decoding unit 806 includes: a first decoding module, used to create a virtual audio decoding object when the above-mentioned target communication type is a voice communication type; configure decoding parameters for the above-mentioned virtual audio decoding object; and decode the above-mentioned target encoded data through the above-mentioned virtual audio decoding object according to the decoding method indicated by the above-mentioned decoding parameters to obtain target audio data.

[0215] Optionally, the above-mentioned decoding unit 806 includes: a second decoding module, which is used to decompress the above-mentioned target encoded data to obtain reference encoded data when the above-mentioned target communication type is a video communication type; determine the encoding type of the above-mentioned reference encoded data, and decode the above-mentioned reference encoded data according to a video decoder matching the above-mentioned encoding type to obtain target video data.

[0216] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0217] According to another aspect of the embodiments of the present invention, an electronic device for implementing the above-mentioned method for acquiring media data is also provided. The electronic device may be Fig. 9 The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig. 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0218] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0219] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0220] S1, in response to receiving a communication request sent by a first client running on a Linux system, determining that a second client establishes a communication connection with the first client;

[0221] S2, receiving the target coded data sent by the first client;

[0222] S3, decoding the target coded data according to the target communication type corresponding to the communication request to obtain target media data;

[0223] S4, playing the target media data.

[0224] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig. 9 The electronic device and the electronic equipment described above are not limited in structure. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations shown.

[0225] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the media data acquisition method and device in the embodiment of the present invention. The processor 904 executes various functional applications and the acquisition of media data by running the software programs and modules stored in the memory 902, that is, the above-mentioned media data acquisition method is realized. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely arranged relative to the processor 904, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically, but not limited to, used to store information such as various elements in the scene screen, data acquisition information, etc. As an example, such as Fig. 9 As shown, the memory 902 may include but is not limited to the determination unit 802, the receiving unit 804, the decoding unit 806 and the playback unit 808 in the display device of the virtual scene. In addition, other module units in the display device of the virtual scene may also be included but are not limited to, which will not be repeated in this example.

[0226] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0227] In addition, the electronic device further includes: a display 908 for displaying a virtual scene in an interface; and a connection bus 910 for connecting various module components in the electronic device.

[0228] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, a peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0229] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0230] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0231] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for acquiring media data.

[0232] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0233] S1, in response to receiving a communication request sent by a first client running on a Linux system, determining that a second client establishes a communication connection with the first client;

[0234] S2, receiving the target coded data sent by the first client;

[0235] S3, decoding the target coded data according to the target communication type corresponding to the communication request to obtain target media data;

[0236] S4, playing the target media data.

[0237] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0238] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to perform all or part of the steps of the above methods of various embodiments of the present invention.

[0239] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0240] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0241] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0243] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for transmitting media data, characterized in that: include: In response to a communication instruction received in a first client running on a Linux system, sending a communication request to a second client, wherein the communication request is used to establish a communication connection between the first client and the second client; In a case where the first client and the second client establish the communication connection, determining a target data interface according to a target communication type indicated by the communication instruction; Acquire target media data received from the first client through the target data interface, and encode the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type; The target encoded data is sent to the second client.

2. The method according to claim 1, characterized in that In the case where the first client and the second client establish the communication connection, determining the target data interface according to the target communication type indicated by the communication instruction includes: In a case where the target communication type indicated by the communication instruction is a voice communication type, determining an audio data monitoring interface as the target data interface; In a case where the target communication type indicated by the communication instruction is a video communication type, a video data acquisition interface is determined as the target data interface.

3. The method according to claim 2, characterized in that When the target communication type indicated by the communication instruction is a voice communication type, determining the audio data monitoring interface as the target data interface includes: Creating a target audio service, wherein the target audio service is used to provide a plurality of data interfaces, each of the data interfaces is used to perform a corresponding data operation; The audio data monitoring interface among the multiple data interfaces is determined as the target data interface, wherein the audio data monitoring interface calls an audio monitoring module in the target audio service, and the audio monitoring module is used to collect audio data in the Linux system.

4. The method according to claim 3, characterized in that The acquiring, through the target data interface, the target media data received from the first client comprises: Register the target callback function; When the audio data received in the first client is detected, the audio data monitoring interface is called through the target callback function to obtain the audio data in the Linux system collected by the audio monitoring module.

5. The method according to claim 2, characterized in that: When the target communication type indicated by the communication instruction is a video communication type, determining the video data acquisition interface as the target data interface includes: Acquire a function interface set provided by a video device driver in the Linux system; The video data acquisition function in the function interface set is determined as the target data interface.

6. The method according to claim 5, characterized in that After obtaining the function interface set provided by the video device driver in the Linux system, the method further includes: Opening a virtual video device in the Linux system through a file reading function in the function interface concentration, wherein the virtual video device is used to obtain video data received from the first client; Acquire a device parameter set of the virtual video device through a parameter reading function in the function interface set, wherein the device parameter set is used to indicate a video data acquisition capability of the virtual video device; According to the device parameter set, configuring target device parameters for the virtual video device through a parameter configuration function in the function interface set; The video data collected by the target virtual video device is obtained through the video data collection function, wherein the target virtual video device is the virtual video device configured according to the target device parameters.

7. The method according to claim 1, characterized in that The encoding process of the target media data to obtain target encoded data comprises: In the case where the target media data is target audio data, creating a virtual audio encoder; encoding the target media data by the virtual audio encoder to obtain the target encoded data; In the case where the target media data is target video data, each video frame in the target video data is traversed, pixel information of the video frame is converted from a first color space to a second color space to obtain reference video data; and encoding processing is performed on the reference video data to obtain the target encoded data.

8. The method according to claim 7, characterized in that In the case where the target media data is target audio data, before creating the virtual audio encoder, at least one of the following is further included: Dynamically adjusting the volume value corresponding to each frame of audio data in the target audio data; Performing noise suppression processing on noise data in the target audio data; In the case where an echo segment is detected in the target audio data, suppression processing is performed on the echo segment, wherein the echo segment is an audio segment that is repeated in the audio data.

9. The method according to claim 7, characterized in that: Before converting the pixel information of the video frame from the first color space to the second color space to obtain the reference video data, the method further includes at least one of the following: Traversing each of the video frames in the target video data, performing filtering processing on a pixel matrix corresponding to the video frame, to obtain a target video frame, wherein the pixel matrix includes pixel values ​​corresponding to each pixel point in the video frame, and a first data amount of high-frequency noise data included in the target video frame is higher than a second data amount of high-frequency noise data included in the video frame before processing; A plurality of continuous video frames are obtained from the target video data; and according to the comparison results of the pixel values ​​of the plurality of continuous video frames, the pixel matrices corresponding to the plurality of continuous video frames are filtered to obtain a plurality of continuous target video frames, wherein a third data amount of the low-frequency noise data included in the plurality of continuous target video frames is higher than a fourth data amount of the low-frequency noise data included in the plurality of continuous video frames before the processing.

10. A method for acquiring media data, characterized in that: include: In response to receiving a communication request sent by a first client running on a Linux system, determining that a second client establishes a communication connection with the first client; Receiving target coded data sent by the first client; Decoding the target coded data according to the target communication type corresponding to the communication request to obtain target media data; Play the target media data.

11. The method according to claim 10, characterized in that The decoding of the target coded data according to the target communication type corresponding to the communication request to obtain the target media data includes: In the case where the target communication type is a voice communication type, creating a virtual audio decoding object; Configuring decoding parameters for the virtual audio decoding object; The target encoded data is decoded by the virtual audio decoding object according to the decoding method indicated by the decoding parameter to obtain target audio data.

12. The method according to claim 10, characterized in that The decoding of the target coded data according to the target communication type corresponding to the communication request to obtain the target media data includes: In the case where the target communication type is a video communication type, decompressing the target coded data to obtain reference coded data; The coding type of the reference coded data is determined, and the reference coded data is decoded according to a video decoder matching the coding type to obtain target video data.

13. A media data transmission device, characterized in that: include: A first sending unit, configured to send a communication request to a second client in response to a communication instruction received in a first client running on a Linux system, wherein the communication request is used to establish a communication connection between the first client and the second client; a determining unit, configured to determine a target data interface according to a target communication type indicated by the communication instruction when the first client and the second client establish the communication connection; an encoding unit, configured to obtain target media data received from the first client through the target data interface, and encode the target media data to obtain target encoded data, wherein the data type of the target media data matches the target communication type; The second sending unit is used to send the target encoded data to the second client.

14. A device for acquiring media data, characterized in that: include: a determining unit, configured to determine, in response to receiving a communication request sent by a first client running on a Linux system, that a second client establishes a communication connection with the first client; A receiving unit, configured to receive target coded data sent by the first client; A decoding unit, configured to decode the target coded data according to the target communication type corresponding to the communication request to obtain target media data; The playing unit is used to play the target media data.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 or 10 to 12 when executed.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 or 10 to 12 are implemented.

17. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 9 or 10 to 12 through the computer program.

Images