Audio and video data transmission method and device, equipment and storage medium

Abstract

The invention provides an audio and video data transmission method and device, equipment and a storage medium, and the method comprises the steps: transmitting a conference instruction to an ARM processor when the conference instruction is received; receiving terminal network information which is sent by the ARM processor and is connected with the articulated naturality web vhub equipment; configuring network connection information between the terminal and the terminal based on the terminal network information; and when the audio and video data are received, sending the audio and video data to the corresponding terminal based on the configured network connection information to realize the articulated naturality web conference. According to the invention, on the basis of the articulated naturality web vhub device, the conference instruction is sent to the ARM processor for processing, the audio and video data is sent to the FPGA for processing, the FPGA and the ARM processor cooperate with each other, transmission of the audio and video data and the conference instruction of the articulated naturality web conference is realized, the transmission bandwidth and the transmission efficiency of the articulated naturality web vhub device are improved, and the conference experience is also improved.

Description

Technical Field

[0001] The present invention relates to the field of visual networking technology, and in particular to a method, device, equipment and computer-readable storage medium for audio and video data transmission. Background Art

[0002] With the development of smart technology, smart conferences are replacing traditional conferences, making conferences more efficient and flexible. Conferences based on the visual networking 10G network tunnel unit (vhub) device make meetings more flexible and smooth. The visual networking vhub device is an intermediate conversion device that connects the visual networking terminal to the visual networking in the external Internet environment. The vhub device can also be understood as a device used to convert the data message protocol of the visual networking and IP network.

[0003] In the related technology, the visual network vhub device is based on the ARM processor to parse and send audio and video data or control instructions. Since the ARM processor's transmission and reception are limited by the CPU processing speed and network bandwidth, the original visual network vhub device is difficult to reach a bandwidth of more than one gigabit, which affects the transmission efficiency and reduces the conference experience.

[0004] Therefore, how to improve the transmission efficiency of the visual network vhub device and enhance the conference experience is a technical problem that needs to be solved. Summary of the invention

[0005] The present invention provides an audio and video data transmission method, device, equipment and computer-readable storage medium to at least solve the technical problem in the related art that the transmission efficiency of the visual networking vhub device is reduced, affecting the conference experience, because the transmission and reception of the ARM processor is limited by the CPU processing speed and network bandwidth. The technical solution of the present invention is as follows:

[0006] According to a first aspect of an embodiment of the present invention, a method for transmitting audio and video data is provided, the method being applied to a visual networking system, the method comprising:

[0007] When receiving a conference instruction, sending the conference instruction to the ARM processor;

[0008] Receiving the terminal network information connected to the visual networking vhub device sent by the ARM processor;

[0009] Configuring network connection information between itself and the terminal based on the terminal network information;

[0010] When audio and video data are received, the audio and video data are sent to the corresponding terminal based on the configured network connection information to realize a video network conference.

[0011] Optionally, before receiving the conference instruction, the method further includes:

[0012] Receiving initialization network configuration information sent by the ARM processor;

[0013] Based on the initialization network configuration information, configure its own initialization network state.

[0014] Before receiving the conference instruction, the method further includes:

[0015] A data packet is received through the visual networking vhub device;

[0016] The data packet is parsed to obtain a data stream, wherein the data stream includes: audio and video data or conference instructions.

[0017] According to a second aspect of an embodiment of the present invention, a method for transmitting audio and video data is provided, the method being applied to a visual networking system, the method comprising:

[0018] Upon receiving the conference command sent by the FPGA, obtaining the terminal network information connected to the visual networking vhub device;

[0019] The terminal network information is sent to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize a video network conference.

[0020] Optionally, before receiving the conference instruction sent by the FPGA, the method further includes:

[0021] In response to the power-on startup of the visual networking vhub device, initialization network configuration information is sent to the FPGA, so that the FPGA configures its own initialization network state based on the initialization network configuration information.

[0022] Optionally, when receiving the conference instruction sent by the FPGA, obtaining the terminal network information connected to the visual networking vhub device includes:

[0023] When receiving the conference command sent by the FPGA, a link is established with the terminal through the visual network vhub device;

[0024] Terminal network information of the terminal is acquired based on the link.

[0025] According to a third aspect of an embodiment of the present invention, there is provided an audio and video data transmission device, the device being applied to a visual networking system, the device comprising:

[0026] A first sending module, configured to send the conference instruction to the ARM processor when receiving the conference instruction;

[0027] A first receiving module is used to receive the terminal network information connected to the visual networking vhub device sent by the ARM processor;

[0028] A first configuration module, configured to configure network connection information between itself and the terminal based on the terminal network information;

[0029] The second sending module is used to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving the audio and video data, so as to realize the video network conference.

[0030] Optionally, the device further comprises:

[0031] A second receiving module, used for receiving the initialization network configuration information sent by the ARM processor before the first receiving module receives the conference instruction;

[0032] The second configuration module is used to configure its own initialization network state based on the initialization network configuration information.

[0033] Optionally, the device further comprises:

[0034] A third receiving module is used to receive a data packet through a visual networking vhub device before the first receiving module receives the conference instruction;

[0035] The parsing module is used to parse the data packet to obtain a data stream, wherein the data stream includes: audio and video data or conference instructions.

[0036] According to a fourth aspect of an embodiment of the present invention, there is provided an audio and video data transmission device, the device being applied to a visual networking system, the device comprising:

[0037] An acquisition module is used to obtain the terminal network information connected to the visual network vhub device when receiving the conference command sent by the FPGA;

[0038] The first sending module is used to send the terminal network information to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize the video network conference.

[0039] Optionally, the device further comprises:

[0040] The second sending module is used to send initialization network configuration information to the FPGA in response to the power-on startup of the visual networking vhub device before the acquisition module acquires the terminal network information connected to the visual networking vhub device, so that the FPGA configures its own initialization network state based on the initialization network configuration information.

[0041] Optionally, the acquisition module includes:

[0042] A connection module is established, which is used to establish a link with the terminal through the visual network vhub device when receiving the conference command sent by the FPGA;

[0043] The information acquisition module is used to acquire the terminal network information of the terminal based on the link established by the connection establishment module.

[0044] According to a fifth aspect of an embodiment of the present invention, a visual networking 10G network tunnel unit vhub device is provided, the visual networking vhub device comprising: a field programmable gate array FPGA and an ARM processor, wherein:

[0045] The ARM processor is used to send initialization network configuration information to the FPGA when the visual networking vhub device is powered on and started;

[0046] The FPGA is used to configure its own initialization network state based on the initialization network configuration information when receiving the initialization network configuration information; and to send the conference instruction to the ARM processor when receiving the conference instruction sent by the conference control software;

[0047] The ARM processor is further used to obtain the terminal network information connected to the visual networking vhub device when receiving the conference instruction sent by the FPGA; and send the terminal network information to the FPGA;

[0048] The FPGA is also used to configure the network connection information between itself and the terminal based on the terminal network information when receiving the terminal network information; and to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving audio and video data, so as to realize a video conference.

[0049] According to a sixth aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0050] processor;

[0051] a memory for storing instructions executable by the processor;

[0052] The processor is configured to execute the instructions to implement the audio and video data transmission method as described above.

[0053] According to a seventh aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio and video data transmission method as described above.

[0054] According to an eighth aspect of an embodiment of the present invention, there is provided a computer program product, comprising a computer program or instructions, which, when executed by a processor of an electronic device, implements the above-mentioned audio and video data transmission method.

[0055] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0056] In an embodiment of the present invention, when receiving a conference instruction, the conference instruction is sent to an ARM processor; the terminal network information connected to the visual network vhub device sent by the ARM processor is received; the network connection information between itself and the terminal is configured based on the terminal network information; and when receiving audio and video data, the audio and video data is sent to the corresponding terminal based on the configured network connection information to realize a visual network conference. That is to say, in an embodiment of the present invention, based on the visual network vhub device, when a conference instruction with a low data volume is received, it is sent to the ARM processor for processing, and the audio and video data with a large data volume is sent to the FPGA for processing. The FPGA obeys the control instructions of the ARM processor, and the two parts cooperate with each other to realize the transmission of audio and video data and conference instructions of the visual network conference, which not only improves the transmission bandwidth and transmission efficiency of the visual network vhub device, but also improves the conference experience.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention, and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a flow chart of an audio and video data transmission method provided by an embodiment of the present invention.

[0060] Figure 2 It is a flow chart of an audio and video data transmission method provided by an embodiment of the present invention.

[0061] Figure 3 It is a block diagram of an audio and video data transmission device provided by an embodiment of the present invention.

[0062] Figure 4 It is another block diagram of an audio and video data transmission device provided by an embodiment of the present invention.

[0063] Figure 5 This is another block diagram of an audio and video data transmission device provided by an embodiment of the present invention.

[0064] Figure 6 It is a block diagram of a visual networking vhub device provided by an embodiment of the present invention.

[0065] Figure 7 This is an application example diagram of a visual networking vhub device provided in an embodiment of the present invention.

[0066] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0067] Fig. 9 The invention is a block diagram of a device for transmitting audio and video data provided by an embodiment of the invention. DETAILED DESCRIPTION

[0068] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0070] Figure 1 is a flow chart of a method for transmitting audio and video data provided by an embodiment of the present invention. Figure 1 As shown, the method is applied to a visual networking system and includes the following steps:

[0071] Step 101: When receiving a conference instruction, the conference instruction is sent to an ARM processor.

[0072] Step 102: Receive the terminal network information connected to the visual networking vhub device sent by the ARM processor.

[0073] Step 103: configure network connection information between itself and the terminal based on the terminal network information.

[0074] Step 104: When the audio and video data are received, the audio and video data are sent to the corresponding terminal based on the configured network connection information to realize a video network conference.

[0075] The audio and video data transmission method described in the present invention can be applied to terminals, servers, etc., without limitation herein. The terminal implementation device can be an electronic device such as a smart phone, a laptop computer, a tablet computer, a desktop computer, a personal digital assistant (PDA, Personal Digital Assistant) and a wearable device. The server can be an independent server or a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, content distribution networks, or a big data and artificial intelligence platform, etc., without limitation herein.

[0076] Combine the following Figure 1 , the specific implementation steps of an audio and video data transmission method provided by an embodiment of the present invention are described in detail.

[0077] In step 101, when a conference instruction is received, the conference instruction is sent to an ARM processor.

[0078] In this step, when the field programmable gate array (FPGA) in the visual network vhub device receives the conference instruction sent by the conference control software, it sends the conference instruction to the ARM processor in the visual network vhub device. Based on the received conference instruction, the ARM processor establishes a link with the terminal (such as a visual network terminal, etc.) through the visual network vhub device; then, it obtains the terminal network information (also called terminal network connection information, etc.) connected to the visual network vhub device; and sends the terminal network information to the FPGA.

[0079] In step 102, terminal network information connected to the visual networking vhub device is received and sent by the ARM processor.

[0080] In this step, the FPGA receives the terminal network information connected to the visual networking vhub device sent by the ARM processor, wherein the terminal network information may include: MAC addresses, IP addresses, ports PORT and other information of multiple terminals participating in the visual networking conference, etc. This embodiment is not limited.

[0081] In step 103, the network connection information between itself and the terminal is configured based on the terminal network information.

[0082] In this step, the FPGA configures the network connection information between itself and the terminal based on the terminal network information. It should be noted that configuring the network connection information between itself and the terminal may be configuring the network connection information between itself and multiple terminals participating in the video network conference.

[0083] In step 104, when the audio and video data are received, the audio and video data are sent to the corresponding terminal based on the configured network connection information to realize the video network conference.

[0084] In this step, when the FPGA receives the audio and video data through the visual network vhub device, it sends the audio and video data to the corresponding terminal based on the configured network connection information to realize the visual network conference.

[0085] Among them, FPGA has the functions of hardware circuit repeatable construction and concurrent processing, and is widely used in interface protocol processing. The embodiment of the present invention is based on the newly added FPGA in the visual networking vhub device to transmit audio and video data with a large amount of data, which is equivalent to increasing the transmission bandwidth in disguise.

[0086] In an embodiment of the present invention, when receiving a conference instruction, the conference instruction is sent to an ARM processor; the terminal network information connected to the visual network vhub device sent by the ARM processor is received; the network connection information between itself and the terminal is configured based on the terminal network information; and when receiving audio and video data, the audio and video data is sent to the corresponding terminal based on the configured network connection information to realize a visual network conference. That is to say, in an embodiment of the present invention, based on the FPGA in the visual network vhub device, the received conference instruction with a low data volume is sent to the ARM processor for processing, and the received audio and video data with a large data volume is processed by itself. The FPGA obeys the control instructions of the ARM processor, and the two parts cooperate with each other to realize the transmission of audio and video data and conference instructions of the visual network conference, which not only improves the transmission bandwidth and transmission efficiency of the visual network vhub device, but also improves the conference experience.

[0087] Optionally, in another embodiment, based on the above embodiment, before receiving the conference instruction, the method may further include:

[0088] The FPGA receives the initialization network configuration information sent by the ARM processor; and configures its own initialization network state based on the initialization network configuration information.

[0089] In an embodiment of the present invention, the ARM processor sends initialization network configuration information to the FPGA in response to the power-on startup of the visual networking vhub device, and the FPGA configures its own initialization network state based on the received initialization network configuration information.

[0090] Optionally, in another embodiment, based on the above embodiment, before receiving the conference instruction, the method also includes: FPGA receives a data packet through a visual network vhub device; and parses the data packet to obtain a data stream, and the data stream includes: audio and video data or conference instructions.

[0091] That is to say, when the visual network vhub device receives a data packet (or a data stream, the data stream includes multiple continuous data packets, etc.), it performs protocol analysis on the data packet to obtain a data stream, which may include audio and video data or conference instructions, such as UDP audio and video data, or TCP conference control instructions, etc. Specifically, the visual network vhub device can perform classified forwarding processing based on the format of the parsed data stream.

[0092] It should be noted that, in this embodiment, it can also be divided according to the size of the received data packet. If the size of the received data packet is within the set range, it means that the data packet is an audio or video data packet, wherein the set range can be 1093-1096 bytes, but is not limited to this, and can also be other ranges, which is not limited in this embodiment; if the size of the data packet is other bytes, it means that the data packet is a conference instruction, for example, 100, 324 or 328 bytes.

[0093] See also Figure 2 , is another flow chart of an audio and video data transmission method provided by an embodiment of the present invention, the method is applied to a visual networking system, and includes:

[0094] Step 201: When receiving the conference command sent by the FPGA, obtain the terminal network information connected to the visual network vhub device.

[0095] In this step, the FPGA receives a data packet based on the visual networking vhub device, and parses the data packet to obtain a data stream, and determines that it is a conference instruction based on the format of the data stream, and then sends the conference instruction to the ARM processor, that is, the ARM processor receives the conference instruction sent by the FPGA, and then can establish a link with the terminal through the visual networking vhub device; based on the link, the terminal network information of the terminal is obtained. Among them, the terminal network information can include multiple terminals participating in the visual networking conference, as well as the MAC address, IP address, port number PORT (or interface, etc.) and other information of each terminal.

[0096] Step 202: Send the terminal network information to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize a video network conference.

[0097] In this step, the ARM processor sends the terminal network information to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize the video network conference.

[0098] Among them, FPGA has the functions of hardware circuit repeatable construction and concurrent processing, and is widely used in interface protocol processing. The embodiment of the present invention is based on the newly added FPGA in the visual networking vhub device to transmit audio and video data with a large amount of data, which is equivalent to increasing the transmission bandwidth in disguise.

[0099] In an embodiment of the present invention, upon receiving a conference instruction sent by an FPGA, the terminal network information connected to the visual network vhub device is obtained; the terminal network information is sent to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information, thereby realizing a visual network conference. In other words, the embodiment of the present invention is based on obtaining the terminal network information connected to the visual network vhub device upon receiving a conference instruction with a low data volume sent by the FPGA; the terminal network information is sent to the FPGA, so that the FPGA sends the audio and video data with a large data volume based on the terminal network information, and the audio and video data and conference instructions of the visual network conference are transmitted and processed through different devices, which not only improves the transmission bandwidth and transmission efficiency of the visual network vhub device, but also improves the conference experience.

[0100] Optionally, in another embodiment, based on the above embodiment, before obtaining the terminal network information connected to the visual network vhub device, the method may also include: in response to the power-on startup of the visual network vhub device, sending initialization network configuration information to the FPGA, so that the FPGA configures its own initialization network state based on the received initialization network configuration information.

[0101] In an embodiment of the present invention, the ARM processor sends initialization network configuration information to the FPGA in response to the power-on startup of the visual networking vhub device, and the FPGA configures its own initialization network state based on the received initialization network configuration information.

[0102] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0103] See also Figure 3 , is a block diagram of an audio and video data transmission device provided by an embodiment of the present invention. The device is applied to a visual networking system, and the device includes: a first sending module 301, a first receiving module 302, a first configuration module 303 and a second sending module 304, wherein:

[0104] The first sending module 301 is used to send the conference instruction to the ARM processor when receiving the conference instruction;

[0105] The first receiving module 302 is used to receive the terminal network information connected to the visual networking vhub device sent by the ARM processor;

[0106] A first configuration module 303, configured to configure network connection information between itself and the terminal based on the terminal network information;

[0107] The second sending module 304 is used to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving the audio and video data, so as to realize the video network conference.

[0108] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0109] A second receiving module, used for receiving the initialization network configuration information sent by the ARM processor before the first receiving module receives the conference instruction;

[0110] The second configuration module is used to configure its own initialization network state based on the initialization network configuration information.

[0111] Optionally, in another embodiment, based on the above embodiment, the device further includes:

[0112] A third receiving module, configured to receive a data packet through the vhub device before the first receiving module receives the conference instruction;

[0113] The parsing module is used to parse the data packet to obtain a data stream, wherein the data stream includes: audio and video data or conference instructions.

[0114] See also Figure 4 , is a block diagram of an audio and video data transmission device provided in an embodiment of the present invention, the device is applied to a visual networking system, the device comprises: an acquisition module 401 and a first sending module 402, wherein,

[0115] The acquisition module 401 is used to acquire the terminal network information connected to the visual networking vhub device when receiving the conference instruction sent by the FPGA;

[0116] The first sending module 402 is used to send the terminal network information to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize a video network conference.

[0117] Optionally, in another embodiment, based on the above embodiment, the device may further include: a second sending module 501, whose structural block diagram is as follows: Figure 5 As shown,

[0118] The second sending module 501 is used for sending initialization network configuration information to the FPGA in response to the power-on startup of the visual networking vhub device before the acquisition module 401 acquires the terminal network information connected to the visual networking vhub device, so that the FPGA configures its own initialization network state based on the initialization network configuration information.

[0119] Optionally, in another embodiment, based on the above embodiment, the acquisition module includes:

[0120] A connection module is established, which is used to establish a link with the terminal through the visual network vhub device when receiving the conference command sent by the FPGA;

[0121] The information acquisition module is used to acquire the terminal network information of the terminal based on the link established by the connection establishment module.

[0122] See also Figure 6 , a block diagram of a visual networking 10G network tunnel unit vhub device is provided in an embodiment of the present invention, wherein the visual networking vhub device comprises: an ARM processor 601 and a field programmable gate array FPGA602, wherein:

[0123] The ARM processor 601 is used to send initialization network configuration information to the FPGA when the visual networking vhub device is powered on and started;

[0124] The FPGA 602 is used to configure its own initialization network state based on the initialization network configuration information when receiving the initialization network configuration information; and send the conference instruction to the ARM processor 601 when receiving the conference instruction sent by the conference control software;

[0125] The ARM processor 601 is also used to obtain the terminal network information connected to the visual networking vhub device when receiving the conference instruction sent by the FPGA; and send the terminal network information to the FPGA502;

[0126] The FPGA602 is also used to configure the network connection information between itself and the terminal based on the terminal network information when receiving the terminal network information; and to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving audio and video data, so as to realize a video conference.

[0127] The visual networking vhub device is applied to the visual networking system.

[0128] Optionally, an embodiment of the present invention further provides an electronic device, including:

[0129] processor;

[0130] a memory for storing instructions executable by the processor;

[0131] The processor is configured to execute the instructions to implement the audio and video data transmission method as described above.

[0132] Optionally, an embodiment of the present invention further provides a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio and video data transmission method as described above.

[0133] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which implement the above-mentioned audio and video data transmission method when executed by a processor of an electronic device.

[0134] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0136] For easier understanding, see also Figure 7 , is an application example diagram of a visual networking vhub device provided in an embodiment of the present invention.

[0137] like Figure 7 As shown, it includes a conference terminal (a conference terminal in this example is taken as an example) and a visual network vhub device, wherein the visual network vhub device includes: an ARM processor and an FPGA, wherein the ARM processor and the FPGA interact with each other in conference commands, and the conference control software is connected to the FPGA through the visual network vhub device. The ARM processor forwards the visual network protocol conference control command (such as TCP conference control command) sent by the conference control software through the FPGA in the visual network vhub device, and then sends the TCP conference control command to the conference terminal through the visual network vhub device. After that, the FPGA receives the command response sent by the conference terminal through the visual network vhub device, and forwards the command response to the conference control software; and when the FPGA receives audio and video data (such as UDP audio and video data) through the visual network vhub device, the UDP audio and video data is sent to the conference terminal through the visual network vhub device. Similarly, the audio and video data (such as UDP audio and video data) sent by the conference terminal will also be received for forwarding and other operations.

[0138] The specific implementation process includes: the ARM processor is used to send initialization network configuration information to the FPGA when the visual network vhub device is powered on and started; the FPGA is used to configure its own initialization network state based on the initialization network configuration information when receiving the initialization network configuration information; and when receiving a conference instruction sent by the conference control software through the visual network vhub device, the conference instruction is sent to the ARM processor; the ARM processor is also used to obtain the terminal network information connected to the visual network vhub device when receiving the conference instruction sent by the FPGA; and send the terminal network information to the FPGA; the FPGA is also used to configure its own network connection information with the terminal based on the terminal network information when receiving the terminal network information; and when receiving audio and video data, the audio and video data is sent to the corresponding terminal based on the configured network connection information to realize a visual network conference.

[0139] Among them, the terminal network information takes two participating terminals (i.e., visual network terminals) as an example, terminal number 1 and terminal number 2, wherein terminal number 1 includes: MAC address (B8EA...), IP address (192.1...), PORT (9999) and other information; terminal number 2 includes: MAC address (B83A...), IP address (192.5...), PORT (8888) and other information.

[0140] That is to say, in this embodiment, the visual network vhub device will send the received visual network audio and video data to the FPGA for processing, and will enter the received conference control software instructions into the FPGA for processing. The ARM processor will configure the network information to the FPGA. Overall, the visual network vhub can use the IP network to connect to the visual network terminal (i.e., the conference terminal), and the visual network terminal can conduct a visual network conference through the interaction of the vhub.

[0141] In this embodiment, the visual network vhub device is powered on and started, and the ARM processor configures the FPGA to initialize the network status. After the initialization is completed, if the FPGA receives a data packet through the visual network vhub device, the data packet is parsed to obtain a conference instruction, which is sent by the conference control software. The conference instruction is sent to the ARM processor. When the ARM processor receives the conference instruction, it sends the conference instruction to the terminal, so that the terminal attempts to connect to the visual network vhub device. When the terminal is successfully connected to the visual network vhub device, the ARM processor obtains the terminal network information of the linked terminal, and sends the obtained terminal network information to the FPGA. The FPGA receives the terminal network information sent by the ARM processor and configures its own network connection information with the terminal. When the FPGA receives audio and video data through the visual network vhub device, it directly sends the audio and video data to the corresponding terminal based on the configured network connection information.

[0142] Figure 8 800 is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0143] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0144] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0145] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0146] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0147] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0148] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0149] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0150] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0151] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0152] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the audio and video data transmission method shown above.

[0153] In an embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device 800 can execute the audio and video data transmission method shown above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0154] In an embodiment, a computer program product is also provided, including a computer program or instructions. When the computer program or instructions are executed by the processor 820 of the electronic device 800, the electronic device 800 executes the above-mentioned audio and video data transmission method.

[0155] Fig. 9 is a block diagram of a device 900 for audio and video data transmission provided by an embodiment of the present invention. For example, the device 900 may be provided as a server. Fig. 9 , the apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the above method.

[0156] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0157] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0158] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for transmitting audio and video data, characterized in that: The method is applied to a visual networking system, and the method comprises: When receiving a conference instruction, sending the conference instruction to the ARM processor; Receiving the terminal network information connected to the visual networking vhub device sent by the ARM processor; Configuring network connection information between itself and the terminal based on the terminal network information; When audio and video data are received, the audio and video data are sent to the corresponding terminal based on the configured network connection information to realize a video network conference.

2. The method for transmitting audio and video data according to claim 1, characterized in that: Before receiving the conference instruction, the method further includes: Receiving initialization network configuration information sent by the ARM processor; Based on the initialization network configuration information, configure its own initialization network state.

3. The method for transmitting audio and video data according to claim 1 or 2, characterized in that: Before receiving the conference instruction, the method further includes: Data packets are received through the visual networking vhub device; The data packet is parsed to obtain a data stream, wherein the data stream includes: audio and video data or conference instructions.

4. A method for transmitting audio and video data, characterized in that: The method is applied to a visual networking system, and the method comprises: When receiving the conference command sent by the FPGA, obtain the terminal network information connected to the visual network vhub device; The terminal network information is sent to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize a video network conference.

5. The method for transmitting audio and video data according to claim 4, characterized in that: Before obtaining the terminal network information connected to the visual networking vhub device, the method further includes: In response to the power-on startup of the visual networking vhub device, initialization network configuration information is sent to the FPGA, so that the FPGA configures its own initialization network state based on the received initialization network configuration information.

6. The method for transmitting audio and video data according to claim 5, characterized in that: When receiving the conference command sent by the FPGA, obtaining the terminal network information connected to the visual networking vhub device includes: When receiving the conference command sent by FPGA, a link is established with the terminal through the visual network vhub device; Terminal network information of the terminal is acquired based on the link.

7. An audio and video data transmission device, characterized in that: The device is applied to a visual networking system, and the device comprises: A first sending module, configured to send the conference instruction to the ARM processor when receiving the conference instruction; A first receiving module is used to receive the terminal network information connected to the visual networking vhub device sent by the ARM processor; A first configuration module, configured to configure network connection information between itself and the terminal based on the terminal network information; The second sending module is used to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving the audio and video data, so as to realize the video network conference.

8. An audio and video data transmission device, characterized in that: The device is applied to a visual networking system, and the device comprises: An acquisition module is used to obtain the terminal network information connected to the visual network vhub device when receiving the conference command sent by the FPGA; The first sending module is used to send the terminal network information to the FPGA, so that when the FPGA receives the audio and video data, it sends the audio and video data to the corresponding terminal based on the terminal network information to realize the video network conference.

9. A 10G network tunnel unit vhub device for visual networking, characterized in that: The visual networking vhub device includes: a field programmable gate array FPGA and an ARM processor, wherein: The ARM processor is used to send initialization network configuration information to the FPGA when the visual networking vhub device is powered on and started; The FPGA is used to configure its own initialization network state based on the initialization network configuration information when receiving the initialization network configuration information; and to send the conference instruction sent by the conference control software to the ARM processor when receiving the conference instruction; The ARM processor is further used to obtain the terminal network information connected to the visual networking vhub device when receiving the conference instruction sent by the FPGA; and send the terminal network information to the FPGA; The FPGA is also used to configure the network connection information between itself and the terminal based on the terminal network information when receiving the terminal network information; and to send the audio and video data to the corresponding terminal based on the configured network connection information when receiving audio and video data, so as to realize a video conference.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio and video data transmission method according to any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio and video data transmission method as described in any one of claims 1 to 6.

Images