An audio and video transmission system based on multimodal processing

Abstract

The present invention relates to the technical field of audio and video transmission, and in particular to an audio and video transmission system based on multimodal processing, comprising: a port data acquisition module, which is used to acquire information of each user terminal in a current session environment; a test module, which is used to acquire the available transmission threshold of each user terminal through a transmission test; an environment identification module, which is used to acquire the maintenance mode of the current session environment and determine the transmission matching result for each user terminal; a data stream driving module, which is used to determine the transmission stream allocation ratio, determine the compression ratio of each modal data stream, and adjust the maintenance mode based on the session environment to update the compression ratio; the present invention adjusts the audio and video transmission compression mode based on the transmission speed difference and data volume difference between each client of the multimodal data, and can adjust the transmission data volume of the multimodal data of each user terminal in the same session environment through the compression ratio setting, so as to ensure the transmission synchronization of each modal data.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video transmission, and in particular to an audio and video transmission system based on multi-modal processing. Background Art

[0002] Multimodality refers to the multiple ways of expressing data or information. In conversation platforms, multimodality generally includes text, voice, pictures, videos, etc., and due to the real-time nature of conversations, multimodal data must be transmitted synchronously. Therefore, for multimodal conversations with large differences in data volume, how to balance the transmission speed of each mode and ensure the continuity and synchronization of data flow in the conversation puts forward requirements for the conversation platform's multimodal data processing and transmission capabilities.

[0003] Chinese patent publication number: CN115174442A, discloses a multi-modal process data transmission method and device, wherein the method includes: reading the original data stream of the transmission device through the port bottom layer driver; performing performance testing on the port bottom layer driver based on the original data stream to determine whether the port bottom layer driver meets the data transmission requirements; when the port bottom layer driver cannot meet the data transmission requirements, using the user driving process to transmit the original data stream. It can be seen that the above data transmission method cannot coordinate the data transmission differences of each user in the same session environment, which will cause the problem of session discontinuity caused by the difference in data transmission capabilities of each user. Summary of the invention

[0004] To this end, the present invention provides an audio and video transmission system based on multimodal processing, so as to overcome the problem of discontinuous user sessions caused by data transmission differences among users in the same session environment in the prior art.

[0005] To achieve the above object, the present invention provides an audio and video transmission system based on multimodal processing, comprising:

[0006] A port data acquisition module, which is used to obtain information of each user terminal in the current session environment, including the data flow threshold and current data flow of the user terminal;

[0007] A test module, which is connected to the port data acquisition module, and is used to calculate the user flow state difference based on the data flow threshold of each user, and to obtain the available transmission threshold of each user terminal through transmission testing;

[0008] An environment identification module, which is used to obtain the maintenance mode of the current session environment, determine the data flow mode of each user terminal based on the maintenance mode, and determine the transmission matching result for each user terminal based on the available transmission threshold of each user terminal;

[0009] A data stream driving module connected to each of the user terminals, for determining a transmission stream allocation ratio based on the data stream modality of a single user terminal and the corresponding transmission matching result, for determining a compression ratio of each modality data stream based on the transmission stream allocation ratio of each of the user terminals in the current session environment, and for updating the compression ratio based on the adjustment of the maintenance modality of the session environment;

[0010] Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode.

[0011] Furthermore, the test module determines the current available transmission threshold of each user terminal by transmitting the data of the setting mode to each user terminal and according to the reception information fed back by each user terminal;

[0012] Wherein, the setting mode is an audio and video mode.

[0013] Furthermore, the port data acquisition module is used to acquire the data flow threshold according to the historical transmission data of each user terminal or to acquire the data flow threshold of each user according to the transmission test.

[0014] Furthermore, the environment identification module determines the data stream mode of each receiving client according to the acquired maintenance mode of the current session environment and the stimulating user end corresponding to the maintenance mode, including a high upload mode and a high reception mode.

[0015] Furthermore, the environment identification module determines the basic data flow of the session environment according to the available transmission threshold of the stimulating user terminal, and determines the transmission matching result of each user terminal according to the comparison result of the available transmission threshold of each user terminal with the basic data flow.

[0016] Furthermore, the data stream driving module determines the transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result, including a high upload ratio and a high reception ratio.

[0017] Furthermore, the data stream driving module determines the compression ratio of the data stream of the video modality data of a single user terminal based on the transmission stream allocation ratio of the single user terminal in the current session environment.

[0018] Furthermore, the data stream driving module determines the transmission speed of the data stream of the audio modal data of the corresponding user terminal based on the compression ratio of the data stream of the video modal data of a single user terminal.

[0019] Furthermore, the data stream driving module is also used to periodically determine the adjustment of the compression ratio of each modal data stream of the corresponding user terminal based on the transmission synchronization rate of the audio and video of a single user terminal in the current session.

[0020] Furthermore, the above-mentioned audio and video transmission system adopts a transmission control method, comprising:

[0021] Obtaining information about each user terminal in the current session environment, including the data flow threshold and current data flow of the user terminal;

[0022] The user flow state difference is calculated based on the data flow threshold of each user, and the available transmission threshold of each user terminal is obtained through transmission test;

[0023] Obtaining the maintenance mode of the current session environment, and determining the data flow mode of each user terminal based on the maintenance mode;

[0024] Determine the transmission matching result for each user terminal based on the available transmission threshold of each user terminal;

[0025] Determine a transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result;

[0026] Determining the compression ratio of each modal data stream based on the transmission stream allocation ratio of each user terminal in the current session environment;

[0027] Based on the adjustment of the maintenance mode of the session environment, the compression ratio is updated to adjust the data stream transmission state in real time;

[0028] Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode.

[0029] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention adjusts the audio and video transmission compression method based on the transmission speed differences and data volume differences between various clients of multimodal data, and can adjust the transmission data volume of multimodal data of each user terminal in the same session environment by setting the compression ratio, thereby ensuring the transmission synchronization of each modal data.

[0030] Furthermore, the present invention updates the compression ratio by adjusting the maintenance mode based on the session environment, and can take into account the role switching of each user terminal in the session and the change of traffic threshold, perform adaptive adjustments, and further ensure the transmission synchronization of each modality data in the same session environment.

[0031] Furthermore, the present invention can reduce the amount of testing by acquiring historical data, reduce the network pressure and equipment pressure on data transmission in the acquisition of user data flow thresholds, further improve the data transmission rate of the session environment, and by acquiring the data flow threshold of each user, it can provide a basis for the subsequent data flow allocation of each user, thereby ensuring the synchronization and real-time transmission of audio and video data of each user in the session environment.

[0032] Furthermore, by judging the data flow mode of each user terminal, the corresponding data transmission mode can be selected according to its different roles and data transmission characteristics in the session environment, avoiding session discontinuity caused by data transmission jams due to redundancy of available transmission bandwidth caused by a single transmission mode or insufficient bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A connection diagram of an audio and video transmission system based on multimodal processing according to the present invention;

[0034] Figure 2 It is a schematic diagram of the implementation process of the audio and video transmission system based on multi-modal processing of the present invention;

[0035] Figure 3 A logical schematic diagram of an audio and video transmission system based on multimodal processing according to the present invention;

[0036] Figure 4 This is a compression logic diagram of the audio and video transmission system based on multimodal processing of the present invention. DETAILED DESCRIPTION

[0037] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0039] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0040] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0041] The conversation environment mentioned in the present invention is a network online environment based on multi-person audio and video conversation scenarios such as corporate training, online meetings, and webinars. It is usually composed of user-end devices and software located at each user, and a conversation environment processing platform and a corresponding processing system located in the cloud. The audio and video transmission system of this embodiment is an implementation of a processing system in the conversation environment processing platform, which is used to process multimodal data such as audio and video in the conversation environment.

[0042] Those skilled in the art should be aware that the complete process of audio and video communication includes the following steps: acquisition, encoding, pre- and post-processing, transmission, decoding, buffering, rendering, etc.

[0043] See also Figure 1 As shown, it is a connection diagram of an audio and video transmission system based on multimodal processing of the present invention. An embodiment of the present invention provides an audio and video transmission system based on multimodal processing, including:

[0044] A port data acquisition module, which is used to obtain information of each user terminal in the current session environment, including a data flow threshold of the user terminal and a current data flow of each user terminal in the current session environment;

[0045] A test module, which is connected to the port data acquisition module, and is used to calculate the user flow state difference based on the data flow threshold of each user, and to obtain the available transmission threshold of each user terminal through transmission testing;

[0046] An environment identification module, which is used to obtain the maintenance mode of the current session environment, determine the data flow mode of each user terminal based on the maintenance mode, and determine the transmission matching result for each user terminal based on the available transmission threshold of each user terminal;

[0047] A data stream driving module connected to each of the user terminals, for determining a transmission stream allocation ratio based on the data stream modality of a single user terminal and the corresponding transmission matching result, for determining a compression ratio of each modality data stream based on the transmission stream allocation ratio of each of the user terminals in the current session environment, and for updating the compression ratio based on the adjustment of the maintenance modality of the session environment;

[0048] Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode.

[0049] The present invention adjusts the audio and video transmission compression method based on the transmission speed differences and data volume differences between the clients of the multimodal data. It can adjust the transmission data volume of the multimodal data of each user terminal in the same session environment by setting the compression ratio, thereby ensuring the transmission synchronization of each modal data.

[0050] Specifically, the test module calculates the user flow state difference based on the data flow threshold of each user.

[0051] Specifically, the user flow state difference is calculated based on the mean square error of the data flow threshold of each user in each current session environment.

[0052] In implementation, the user flow state difference can characterize the difference in data flow transmission speed of the same audio and video data for each user in the current session environment, and characterize the difference in tolerance of the transmission data flow between each user in the session environment, thereby reflecting the difference in data flow speed caused by differences in user devices. It can be understood that the data flow threshold and the current data flow both refer to the transmission rate of the data flow.

[0053] Specifically, the test module transmits data of a set mode to each user terminal and determines the current available transmission threshold of each user terminal according to the reception information fed back by each user terminal;

[0054] Wherein, the setting mode is an audio and video mode.

[0055] In implementation, the test module sends a preset data packet of a set mode to each user terminal, and determines the current available transmission threshold of each user terminal by obtaining the reception timestamp of the preset data packet received by each user terminal and the sending timestamp of the preset data packet of the test module.

[0056] Preferably, the mode is set to include a data packet for implementing synchronous transmission of audio and video, and the size of the preset data packet is determined according to the maximum real-time data transmission rate supported by the session environment, or is set according to the application scenario, but the implementation of the above preset data packet is not limited to this, and those skilled in the art can also adjust the size and content of the preset data packet according to actual needs. As an implementation method, the preset data packet can be an audio and video introduction or usage instructions for the current session environment, and the size of the preset data packet is set to the data flow threshold of each user in the current session environment.

[0057] It is understandable that, since the amount of data in the preset data packet has been obtained in advance, the reception rate of the preset data packet can be calculated by obtaining the reception timestamp of each user terminal receiving the preset data packet and the transmission timestamp of the preset data packet of the test module, and the reception rate is set as the available transmission threshold, which is the transmission rate threshold of each user terminal. Therefore, for the accuracy of the test, the size of the preset data packet should be large, containing at least 300M of data.

[0058] Specifically, the port data acquisition module is used to acquire the data flow threshold according to the historical transmission data of each user terminal or to acquire the data flow threshold of each user according to the transmission test.

[0059] In implementation, the port data acquisition module retrieves historical data and matches it with each user terminal to obtain the maximum value of the data transmission rate of each user terminal in the historical session environment as the data flow threshold, or obtains the current available transmission threshold of the user terminal as the data flow threshold of the user by performing the above-mentioned test (transmitting data of the set mode to each user terminal) for the user terminal without historical data.

[0060] The present invention can reduce the amount of testing by acquiring historical data, reduce the network pressure and equipment pressure on data transmission in acquiring user data flow thresholds, further improve the data transmission rate of the session environment, and provide a basis for subsequent data flow allocation to each user by acquiring the data flow threshold of each user, thereby ensuring the transmission synchronization and real-time performance of audio and video data of each user in the session environment.

[0061] Specifically, the environment identification module determines the data stream mode of each receiving client according to the acquired maintenance mode of the current session environment and the stimulating user end corresponding to the maintenance mode, including a high upload mode and a high reception mode.

[0062] In implementation, the maintenance mode of the session environment can be obtained according to the creation information of the session environment, or determined according to the transmission status of the audio and video data in the current session environment. Specifically, if there is a user end uploading or receiving audio and video data in the current session environment, it is considered that the maintenance mode of the current session environment is the audio and video mode; if there is no user end uploading or receiving video data in the current session environment, it is considered that the current maintenance mode is the audio mode;

[0063] At this time, the user end that uploads audio or video or audio and video data in the session environment is determined as the exciting user end, and its data flow mode is determined to be a high upload mode; the user end that only receives audio or video or audio and video data in the session environment is determined as a receiving user end, and its data flow mode is determined to be a high receiving mode.

[0064] By judging the data flow mode of each user terminal, the corresponding data transmission method can be selected according to its different roles and data transmission characteristics in the session environment, avoiding session discontinuity caused by redundancy of available transmission bandwidth caused by a single transmission method or data transmission packet loss and jamming caused by insufficient bandwidth.

[0065] Specifically, the environment identification module determines the basic data flow of the session environment based on the available transmission threshold of the stimulating user terminal, and determines the transmission matching result of each user terminal based on the comparison result of the available transmission threshold of each user terminal with the basic data flow, and the transmission matching result includes transmission matching passed and transmission matching failed.

[0066] In implementation, when there is only one stimulating user terminal, the basic data flow is set to 80% to 90% of the available transmission threshold of the stimulating user terminal. When there are multiple stimulating user terminals, the basic data flow takes the maximum value of the available transmission thresholds of each stimulating user terminal.

[0067] The transmission matching result of each user terminal is determined by comparing the available transmission threshold of each user terminal with the basic data flow to obtain a comparison result.

[0068] In a specific embodiment, if the available transmission threshold of a single user terminal is greater than or equal to the basic data flow, it is determined that the transmission match of the current single user terminal has passed;

[0069] In a specific embodiment, if the available transmission threshold of a single user terminal is less than the basic data flow, it is determined that the current transmission match of the single user terminal fails.

[0070] Specifically, the data stream driving module determines the transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result, including a high upload ratio and a high reception ratio.

[0071] In a specific embodiment, if the data stream mode of a single user terminal is a high upload mode, and the corresponding transmission matching result is a transmission matching pass, it is determined that the transmission stream allocation ratio of the current single user is a high upload ratio;

[0072] In a specific embodiment, if the data stream mode of a single user terminal is a high upload mode, and the corresponding transmission matching result is a transmission matching failure, it is determined that the transmission stream allocation ratio of the current single user is a high receiving ratio;

[0073] It can be understood that the data stream mode of a single user end is a high upload mode, and the corresponding transmission matching result is that the transmission matching fails. At this time, the single user is an exciting user end with a smaller available transmission threshold among several exciting user ends. Since it needs to receive data uploaded by other exciting user ends, the setter is set to a high receiving ratio to ensure that the data reception of the user end avoids freezes caused by network congestion and insufficient bandwidth;

[0074] In a specific embodiment, if the data stream mode of a single user terminal is a high receiving mode, and the corresponding transmission matching result is a transmission matching pass, it is determined that the transmission stream allocation ratio of the current single user is a high receiving ratio;

[0075] In a specific embodiment, if the data stream mode of a single user terminal is a high receiving mode, and the corresponding transmission matching result is a transmission matching failure, it is determined that the transmission stream allocation ratio of the current single user is a high receiving ratio;

[0076] It can be understood that the high upload ratio and the high reception ratio are both set ratio values, which are the ratio of the upload reserved transmission rate to the reception reserved transmission rate. Preferably, the ratio value range of the high upload ratio is set to the upload reserved transmission rate / receive reserved transmission rate = 4 / 1~5 / 1, and the ratio value range of the high reception ratio is set to the upload reserved transmission rate / receive reserved transmission rate = 1 / 3~1 / 5. Those skilled in the art should know that in each of the above situations, when the corresponding transmission matching result is that the transmission matching fails, the higher the ratio value of the high upload ratio is within the value range of the upload reserved transmission rate / receive reserved transmission rate, the smaller the ratio value of the high reception ratio is within the value range of the upload reserved transmission rate / receive reserved transmission rate.

[0077] It is understandable that in a real-time conversation environment, multimodal data uploading and data receiving can occur simultaneously. Therefore, the total data stream bandwidth of each user terminal needs to be allocated for uploading and receiving. Therefore, setting the ratio of the upload reserved transmission rate to the reception reserved transmission rate can determine the bandwidth allocation of the upload and reception data of the corresponding user terminal, thereby ensuring the continuity of the user's current session.

[0078] See also Figure 4 As shown, it is a compression logic diagram of the audio and video transmission system based on multimodal processing of the present invention. Specifically, the data stream driving module determines the compression ratio of the data stream of the video modality data of a single user terminal based on the transmission stream allocation ratio of a single user terminal in the current session environment.

[0079] In implementation, in a specific embodiment, the data stream driving module calculates the corresponding upload reserved transmission rate and the reception reserved transmission rate according to the transmission stream allocation ratio of a single user terminal and the corresponding available transmission threshold, and compares the average of the reception reserved transmission rates of each user terminal with the upload reserved transmission rate of the stimulating user terminal to obtain a comparison result, and determines whether the data stream of the video modality data of the stimulating user terminal needs to be compressed according to the comparison result;

[0080] Specifically, the available transmission threshold of a single stimulating user terminal i is set to Ki, and its transmission flow allocation ratio is upload reserved transmission rate / receive reserved transmission rate = ai / bi, and the average of the receive reserved transmission rate of each user terminal in the session environment is Kbn;

[0081] Then the corresponding upload reserved transmission rate Kai = Ki × ai / (ai + bi);

[0082] Then the corresponding receiving reserved transmission rate Kbi=Ki×bi / (ai+bi);

[0083] Compare Kai and Kbn;

[0084] If Kai ≥ Kbn, it is determined that the data stream of the video modality data that stimulates the user terminal i does not need to be compressed;

[0085] If Kai<Kbn, it is determined that the data stream of the video modal data of the stimulating user terminal i needs to be compressed, and the data stream compression ratio is set as: compressed data packet size / original data packet size=γ×Kai / Kbn, wherein γ is a compression coefficient determined according to the difference in user flow patterns, γ is negatively correlated with the difference in user flow patterns, and γ<1. Preferably, γ takes a value in the range of 0.6 to 0.9. At this time, the original data packet is the original data packet file of the video data of the stimulating user terminal collected by the stimulating user terminal acquisition device and the session environment support software, and what is compressed is the upload data stream of the video modal data of the stimulating user terminal. The compressed data packet will be sent to the transmission processing platform of the session environment, and the compression is used to reduce the upload data volume of the stimulating user terminal, increase the upload speed, and reduce the data stream bandwidth occupied by the uploaded data.

[0086] In another specific embodiment, the data stream driving module compares the receiving reserved transmission rate of the single receiving user end with the compressed audio and video upload rate of the exciting data end, and determines whether the data stream of the video modality data of the single receiving user end needs to be compressed according to the comparison result;

[0087] Specifically, the reserved transmission rate of a single receiving user terminal j is set to Kbj, and the compressed audio and video upload rate of the excitation data terminal is set to Ka.

[0088] Compare Kbj with Ka,

[0089] If Kbj≥Ka, it is determined that the data stream of the video modality data of the receiving user end j does not need to be compressed;

[0090] If Kbj<Ka, it is determined that the data stream of the video modal data of the receiving user terminal j needs to be compressed, and the data stream compression ratio is set as: compressed data packet size / original data packet size=λ×Kai / Kbn, wherein λ is a compression coefficient determined according to the stability of the user network bandwidth, λ is positively correlated with the stability of the user network bandwidth, and λ<1. Preferably, γ takes a value in the range of 0.6~0.9. The higher the stability of the user network bandwidth, the closer the value of λ is to 0.9. At this time, what is compressed is the receiving data stream of the video modal data transmitted from the transmission processing platform of the session environment to the receiving user terminal. The compressed data packet will be sent to the receiving user terminal for data packet decoding. The compression is used to reduce the amount of received data at the receiving user terminal, increase the receiving speed, and reduce the data stream bandwidth occupied by the received data. The stability of the user network bandwidth can be obtained by any method in the prior art, which will not be repeated here.

[0091] It can be understood that the above compression is only performed on video data, and the transmission of audio data is not affected. In multimodal transmission, the amount of audio data is significantly lower than that of video data. Therefore, the original encoding form of the audio data is retained at this time to facilitate the subsequent adjustment of the compression ratio of each modal data stream for the user end according to the synchronization rate of audio and video.

[0092] Specifically, the data stream driving module determines the transmission speed of the data stream of the audio modal data of the corresponding user terminal based on the compression ratio of the data stream of the video modal data of a single user terminal.

[0093] During implementation, after determining the corresponding data stream compression ratio, the data stream driving module compresses the audio and video data received by the transmission processing platform of the session environment in real time and transmits it to the corresponding user terminal. At this time, the real-time video data volume and audio data volume compressed and transmitted to each user terminal can be obtained, and the transmission speed of the data stream of the audio modal data of each user terminal is determined based on the real-time video data volume and audio data volume and the total data stream transmission bandwidth of the current user.

[0094] Specifically, the data stream driving module is also used to periodically determine the adjustment of the compression ratio of each modal data stream of the corresponding user terminal based on the synchronization rate of the audio and video of a single user terminal in the current session.

[0095] In implementation, the data stream driver module periodically obtains the maintenance mode of the current session environment, and determines the audio and video synchronization rate of each user terminal according to the synchronization of the transmission timestamp of the data packet. The transmission synchronization rate of a single mode (audio mode or video mode) is determined according to the mean square error of the reception timestamp corresponding to the time of the simultaneous data packets of the single mode received by each user terminal. The transmission synchronization rate of a single user is determined according to the difference (absolute value) between the reception timestamp corresponding to the time of the reception timestamp of the data packet of the audio mode received by a single user terminal and the reception timestamp corresponding to the time of the data packet of the video mode sent simultaneously.

[0096] When the transmission synchronization rate of a single mode or the transmission synchronization rate of a single user exceeds the corresponding preset synchronization rate preset value, it is determined that the compression ratio of each modal data stream of the corresponding user end needs to be adjusted.

[0097] Specifically, if the single-mode transmission synchronization rate exceeds the preset single-mode synchronization rate preset value, it is determined that the compression ratio of the corresponding modal data stream that stimulates the user end needs to be adjusted, and the compression ratio of the adjusted data stream is reduced, that is, the ratio of the compressed data packet size / the original data packet size is reduced. The specific adjustment amount can be reduced according to the preset compression ratio. Generally, the compression is 80%~85% before the adjustment, which will not be repeated here.

[0098] Specifically, if the transmission synchronization rate of a single user exceeds the preset single-user synchronization rate preset value, it is determined that the compression ratio of the video modal data stream of the corresponding user end needs to be adjusted. The compression ratio of the adjusted data stream is reduced, that is, the ratio of the compressed data packet size / the original data packet size is reduced. The specific adjustment amount can be reduced according to the preset compression ratio. Generally, the compression is 80% to 85% of the pre-adjustment ratio, which will not be repeated here.

[0099] It is understandable that the preset single-mode synchronization rate preset value and the single-user synchronization rate preset value can be independently set according to the conversation environment or according to the needs of the user, and will not be described in detail here.

[0100] Specifically, the data stream driving module updates the compression ratio based on the adjustment of the maintenance mode of the session environment, and determines whether to compress each modality data stream according to the changed maintenance mode after the maintenance mode is changed. Generally, when the maintenance mode is the audio mode, data stream compression is not required.

[0101] Specifically, the data stream driving module updates the compression ratio based on the change of the stimulating user terminal in the session environment. After the stimulating user terminal is changed, the transmission stream allocation ratio is re-determined and the compression ratio of each modal data stream is calculated according to the changed stimulating user terminal.

[0102] See also Figure 2 and Figure 3 As shown, it is a schematic diagram of the implementation process of the audio and video transmission system based on multimodal processing of the present invention and a logical schematic diagram of the audio and video transmission system based on multimodal processing of the present invention. The embodiment of the present invention also provides a transmission control method using the above audio and video transmission system, including:

[0103] Step S1, obtaining information of each user terminal in the current session environment, including a data flow threshold and a current data flow of the user terminal;

[0104] Step S2, calculating the user flow difference based on the data flow threshold of each user, and obtaining the available transmission threshold of each user terminal through transmission test;

[0105] Step S3, obtaining the maintenance mode of the current session environment, and determining the data flow mode of each user terminal based on the maintenance mode;

[0106] Step S4, determining the transmission matching result for each user terminal based on the available transmission threshold of each user terminal;

[0107] Step S5, determining a transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result;

[0108] Step S6, determining the compression ratio of each modal data stream based on the transmission stream allocation ratio of each user terminal in the current session environment;

[0109] Step S7, based on the adjustment of the maintenance mode of the session environment, updating the compression ratio to adjust the data stream transmission state in real time;

[0110] Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode.

[0111] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An audio and video transmission system based on multimodal processing, It is characterized in that include: A port data acquisition module, which is used to obtain information of each user terminal in the current session environment, including the data flow threshold and current data flow of the user terminal; A test module, which is connected to the port data acquisition module, is used to calculate the user flow difference based on the data flow threshold of each user, and to obtain the available transmission threshold of each user terminal through transmission test; the user flow difference can represent the difference in data flow transmission speed of the same audio and video data for each user in the current session environment; An environment identification module is used to obtain the maintenance mode of the current session environment, determine the data flow mode of each user terminal based on the maintenance mode, and determine the transmission matching results for each user terminal based on the available transmission threshold of each user terminal; the maintenance mode of the session environment can be obtained according to the creation information of the session environment, or determined according to the transmission status of the audio and video data in the current session environment; if there is a user terminal uploading or receiving audio and video data in the current session environment, it is considered that the maintenance mode of the current session environment is the audio and video mode; if there is no user terminal uploading or receiving video data in the current session environment, it is considered that the current maintenance mode is the audio mode; the user terminal that uploads audio or video or audio and video data in the session environment is determined as an exciting user terminal, and its data flow mode is determined to be a high upload mode; the user terminal that only receives audio or video or audio and video data in the session environment is determined as a receiving user terminal, and its data flow mode is determined to be a high receiving mode; A data stream driving module connected to each of the user terminals, for determining a transmission stream allocation ratio based on the data stream modality of a single user terminal and the corresponding transmission matching result, for determining a compression ratio of each modality data stream based on the transmission stream allocation ratio of each of the user terminals in the current session environment, and for updating the compression ratio based on the adjustment of the maintenance modality of the session environment; Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode; The data stream driver module periodically obtains the maintenance mode of the current session environment, and determines the audio and video synchronization rate of each user end according to the synchronization of the transmission timestamp of the data packet. The transmission synchronization rate of the audio mode or video mode is determined according to the mean square error of the receiving timestamp corresponding to the time of the simultaneous data packets of the single mode received by each user end. The transmission synchronization rate of a single user is determined according to the difference between the receiving timestamp corresponding to the time of the receiving timestamp of the data packet of the audio mode received by a single user end and the receiving timestamp corresponding to the time of the data packet of the video mode sent simultaneously, and the absolute value is taken.

2. The audio and video transmission system based on multimodal processing according to claim 1, It is characterized in that The test module transmits data of the set mode to each user terminal and determines the current available transmission threshold of each user terminal according to the reception information fed back by each user terminal; Wherein, the setting mode is an audio and video mode.

3. The audio and video transmission system based on multimodal processing according to claim 2, It is characterized in that The port data acquisition module is used to acquire the data flow threshold according to the historical transmission data of each user terminal or to acquire the data flow threshold of each user according to the transmission test.

4. The audio and video transmission system based on multimodal processing according to claim 3, It is characterized in that The environment identification module determines the data stream mode of each receiving client according to the acquired maintenance mode of the current session environment and the stimulating user end corresponding to the maintenance mode, including a high upload mode and a high reception mode.

5. The audio and video transmission system based on multimodal processing according to claim 4, It is characterized in that The environment identification module determines the basic data flow of the session environment according to the available transmission threshold of the stimulating user terminal, and determines the transmission matching result of each user terminal according to the comparison result of the available transmission threshold of each user terminal and the basic data flow.

6. The audio and video transmission system based on multimodal processing according to claim 5, It is characterized in that The data stream driving module determines the transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result, including a high upload ratio and a high reception ratio.

7. The audio and video transmission system based on multimodal processing according to claim 6, It is characterized in that The data stream driving module determines the compression ratio of the data stream of the video modality data of a single user terminal based on the transmission stream allocation ratio of the single user terminal in the current session environment.

8. The audio and video transmission system based on multimodal processing according to claim 7, It is characterized in that The data stream driving module determines the transmission speed of the data stream of the audio modality data of the corresponding user terminal based on the compression ratio of the data stream of the video modality data of a single user terminal.

9. The audio and video transmission system based on multimodal processing according to claim 1, It is characterized in that The data stream driving module is also used to periodically determine the adjustment of the compression ratio of each modal data stream of the corresponding user terminal based on the transmission synchronization rate of the audio and video of a single user terminal in the current session.

10. The audio and video transmission system based on multimodal processing according to any one of claims 1 to 9, It is characterized in that The audio and video transmission system adopts a transmission control method, including: Obtaining information about each user terminal in the current session environment, including the data flow threshold and current data flow of the user terminal; The user flow state difference is calculated based on the data flow threshold of each user, and the available transmission threshold of each user terminal is obtained through transmission test; Obtaining the maintenance mode of the current session environment, and determining the data flow mode of each user terminal based on the maintenance mode; Determine the transmission matching result for each user terminal based on the available transmission threshold of each user terminal; Determine a transmission stream allocation ratio based on the data stream mode of a single user terminal and the corresponding transmission matching result; Determining the compression ratio of each modal data stream based on the transmission stream allocation ratio of each user terminal in the current session environment; Based on the adjustment of the maintenance mode of the session environment, the compression ratio is updated to adjust the data stream transmission state in real time; Among them, the maintenance mode includes real-time audio mode and real-time audio and video mode.

Images