Cross-device and cross-platform voice and video integrated communication method and cross-device and cross-platform voice and video integrated communication system

Abstract

The invention relates to the technical field of communication, and particularly provides a cross-device and cross-platform voice and video integrated communication method and system, and the method comprises the steps: obtaining network state information, device performance information, and operation information, audio information and video information of a user in a communication application; analyzing and acquiring a user communication task intention according to the operation information, the audio information and the video information; obtaining parameter adjustment strategies corresponding to different media streams according to the network state information, the equipment performance information and the user communication task intention; for each media stream, adjusting the parameters of the media stream according to the corresponding parameter adjustment strategy; according to the method, the problem that the audio and video quality cannot meet the requirements of the communication task when the communication task changes due to the fact that a self-adaptive adjustment strategy based on the equipment performance and the network state parameters cannot adapt to the dynamic change of the communication task can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a cross-device and cross-platform voice and video integrated communication method and system. BACKGROUND

[0002] In a cross-device and cross-platform voice and video communication system, in order to cope with complex network environment (such as bandwidth limitation, high delay, packet loss) and device performance difference, the system usually uses adaptive adjustment strategy to dynamically adjust the parameters of audio and video such as code rate, resolution, frame rate and encoding parameters, in order to maximize the quality of audio and video under the premise of ensuring the continuity of communication. For example, in the environment where the user uses a desktop computer through a wired network in an office with good network condition and powerful device performance, the system can provide high-resolution, high-frame-rate video and high-sampling-rate audio. When the user needs to enter the production workshop to demonstrate the exhibits or the processing flow, the user will switch the ongoing communication from the high-performance device to the smartphone carried by the user and carry the smartphone into the production workshop. The communication system will recognize the decrease of device processing capability, screen size, camera and microphone specifications and the fluctuation of network connection quality. At this time, the communication system will use the adaptive adjustment strategy based on device performance and network state parameters to reduce the parameters of each media stream (such as reducing video resolution and codec parameters), in order to maintain the continuity of the call.

[0003] However, the user's communication task is not constant, for example, in the scene of remote fault diagnosis or device inspection, the user may need to use the camera of the smartphone to take real-time photos of the details of a specific part of the faulty device for observation and guidance by the remote expert, that is, the communication task will change from "maintaining voice communication" to "providing clear visual information". The existing adaptive adjustment strategy will uniformly and symmetrically reduce the quality of all media streams (for example, simultaneously reduce the code rate, resolution and frame rate of all streams) when facing the aforementioned network quality decrease and device limitation. Although this "one-size-fits-all" strategy ensures the continuity of the connection, it may cause the resolution and frame rate of the uplink video stream to be excessively compressed, resulting in the inability to identify the details of the specific part through the uplink video stream, and the remote expert cannot provide effective guidance. Therefore, the existing technology has the problem that the adaptive adjustment strategy based on device performance and network state parameters cannot adapt to the dynamic changes of the communication task, resulting in the quality of audio and video being unable to meet the needs of the communication task when the communication task changes, thereby affecting the effectiveness of the communication.

[0004] At present, there is no effective technical solution to the above problems. It should be noted that the above information disclosed in this part is only used to understand the background of the inventive concept, and therefore can contain information that does not constitute prior art. SUMMARY

[0005] The purpose of the present application is to provide a voice and video integrated communication method and system across devices and platforms, which can effectively solve the problem that the quality of voice and video cannot meet the requirements of the communication task when the communication task changes due to the adaptive adjustment strategy based on device performance and network state parameters cannot adapt to the dynamic changes of the communication task.

[0006] In a first aspect, the present application provides a voice and video integrated communication method across devices and platforms, comprising the following steps:

[0007] S1, obtaining network state information, device performance information, and user operation information, audio information and video information in the communication application;

[0008] S2, obtaining user communication task intention according to operation information, audio information and video information;

[0009] S3, obtaining parameter adjustment strategy corresponding to different media streams according to network state information, device performance information and user communication task intention;

[0010] S4, for each media stream, adjusting the parameters of the media stream according to the corresponding parameter adjustment strategy.

[0011] In a second aspect, the present application further provides a voice and video integrated communication system across devices and platforms, comprising:

[0012] An information acquisition module is configured to obtain network state information, device performance information, and user operation information, audio information and video information in the communication application;

[0013] A task intention analysis module is configured to obtain user communication task intention according to operation information, audio information and video information;

[0014] A parameter adjustment strategy determination module is configured to obtain parameter adjustment strategy corresponding to different media streams according to network state information and user communication task intention;

[0015] A parameter adjustment module is configured to adjust the parameters of each media stream according to the corresponding parameter adjustment strategy.

[0016] From the above, the voice and video integrated communication method and system across devices and platforms provided in the application can perceive the current communication task intention of the user by acquiring and analyzing the operation information, audio information and video information of the user in the communication process, then dynamically determine the differentiated parameter adjustment strategy for different media streams by combining the communication task intention with the traditional network state information and device performance information, and finally adjust the parameters of different media streams based on the parameter adjustment strategy. Since the adaptive adjustment strategy of the application can adapt to the dynamic changes of device performance, network state parameters and communication tasks, that is, when the network quality decreases and the device is limited, the application will not uniformly and symmetrically reduce the quality of all media streams, that is, the application will not adopt the "one-size-fits-all" strategy when the network quality decreases and the device is limited. Therefore, the application can effectively solve the problem that the adaptive adjustment strategy based on device performance and network state parameters cannot adapt to the dynamic changes of the communication task, so that the audio and video quality cannot meet the needs of the communication task when the communication task changes, thereby effectively improving the effectiveness of communication. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a voice and video integrated communication method across devices and platforms provided in the application.

[0018] Figure 2 A structural schematic diagram of a voice and video integrated communication system across devices and platforms provided in the application.

[0019] Reference signs: 1, information acquisition module; 2, task intention analysis module; 3, parameter adjustment strategy determination module; 4, parameter adjustment module. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the application.

[0021] It should be noted that similar reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, that item does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0022] The first aspect, as shown in the figure, the present application provides a cross-device and platform voice and video integrated communication method, which comprises the following steps: Figure 1

[0023] S1, obtaining network state information, device performance information, and user operation information, audio information and video information in the communication application;

[0024] S2, obtaining user communication task intention according to operation information, audio information and video information;

[0025] S3, obtaining parameter adjustment strategy corresponding to different media streams according to network state information, device performance information and user communication task intention;

[0026] S4, for each media stream, adjusting the parameters of the media stream according to the corresponding parameter adjustment strategy.

[0027] Step S1 obtains various types of information (network state information, device performance information, and user operation information, audio information and video information in the communication application). The network state information of this embodiment refers to a parameter set that can reflect the state and quality of the current used network. This embodiment can obtain the network state information by monitoring network bandwidth, packet loss rate, network delay and other parameters. The device performance information of this embodiment refers to a parameter set that can reflect the processing capability of the device currently used by the user. This embodiment can obtain the device performance information by querying the processor capability, memory capacity, screen size, camera and microphone specifications and other parameters of the device based on the structure provided by the device operating system. This embodiment can obtain the user operation information in the communication application by recording the interactive behaviors of the user such as screen touch, sliding, scaling, key input, etc. This embodiment can obtain the audio information of the user in the communication application by using the microphone to collect the voice signal of the user. This embodiment can obtain the video information of the user in the communication application by collecting the video pictures input by the camera and the screen sharing content, etc. Step S1 provides basic data for subsequent analysis of user intention and formulation of adjustment strategy by obtaining different types of information.

[0028] ​The user communication task intention of step S2 refers to a specific goal or focus point that the user wants to achieve through audio and video communication in the current communication process. In this embodiment, a rule-based analysis method or a machine learning model can be used to analyze and obtain the user communication task intention according to the operation information, audio information, and video information. Specifically, the flow of step S2 can be as follows: using speech recognition technology to perform speech recognition on the audio information; analyzing specific gesture sequences, input instructions, or interaction modes in the operation information; using image recognition technology to recognize the content of the video information (such as faces and objects), analyze the stability of the video, the focus area, or the characteristics of the video changes; and based on the pre-set task intention analysis rules, comprehensively analyzing the speech recognition results, operation recognition results, and video recognition results to infer the type of communication activity that the user wants to perform (user communication task intention), for example, inferring that the user wants to perform a regular conversation, display specific content, or screen sharing. Step S2 is equivalent to converting the user's behavior, speech, and video into a task intention representation that the system can understand.

[0029] Since the network state information and the device performance information reflect the current communication environment and the device capability limitations, and the user communication task intention can reflect the type of communication activity that the user wants to perform, which is associated with the communication content or quality that the user wants to prioritize (for example, if the type of communication activity that the user wants to perform is to display specific content, then the user wants to prioritize the quality of the uploaded video), step S3 needs to determine the parameter adjustment strategy for different media streams based on the network state information and the device performance information obtained in step S1 and the user communication task intention analyzed in step S2, that is, step S3 is equivalent to considering how to differentially configure the parameters such as the code rate, resolution, frame rate, and encoding method of the uplink audio stream, downlink audio stream, uplink video stream, and downlink video stream, etc. according to the user communication task intention under the current environmental limitations, for example, when the user communication task intention is to display details and the network quality or device capability is limited, the parameter adjustment strategy of this embodiment tends to allocate more resources to the uplink video stream and less resources to the downlink video stream. This embodiment can use the method of consulting a pre-set strategy rule library according to the network state information, device performance information, and user communication task intention to obtain the parameter adjustment strategy corresponding to different media streams. This embodiment can also use the method of executing decision tree logic on the network state information, device performance information, and user communication task intention to obtain the parameter adjustment strategy corresponding to different media streams. This embodiment can also input the network state information, device performance information, and user communication task intention into a machine learning model to obtain the parameter adjustment strategy corresponding to different media streams.

[0030] Step S4 refers to performing specific parameter adjustment operations on each media stream according to the parameter adjustment strategy determined in step S3. In this embodiment, the interface of the audio and video codec and the transmission module is called to set the corresponding parameter value to adjust the parameters of the corresponding media stream according to the parameter adjustment strategy, for example, adjusting the encoding code rate of the uplink video stream to adapt to the network bandwidth, adjusting the resolution of the downlink video stream to adapt to the device screen size, adjusting the sampling rate or encoding mode of the audio stream to adapt to the network delay or user communication task intention (such as voice clarity priority). Step S4 is the execution link of the parameter adjustment strategy to ensure that the system can actually control the parameters of each media stream according to the comprehensive consideration result.

[0031] The core innovation of the present application is that the current communication task intention of the user is perceived by acquiring and analyzing the operation information, audio information and video information of the user in the communication process, and then the communication task intention is combined with the traditional network state information and device performance information to dynamically determine the differentiated parameter adjustment strategy for different media streams, and finally the parameters of different media streams are adjusted based on the parameter adjustment strategy. Since the adaptive adjustment strategy of the present application can adapt to the dynamic changes of device performance, network state parameters and communication tasks, that is, when the network quality decreases and the device is limited, the present application will not uniformly and symmetrically reduce the quality of all media streams, that is, the present application will not adopt the "one-size-fits-all" strategy when the network quality decreases and the device is limited. Therefore, the present application can effectively solve the problem that the adaptive adjustment strategy based on device performance and network state parameters cannot adapt to the dynamic changes of the communication task, resulting in that the audio and video quality cannot meet the requirements of the communication task when the communication task changes, thereby effectively improving the effectiveness of communication.

[0032] Specifically, the method first acquires network state information, device performance information, and user operation information, audio information, and video information in the communication application to provide comprehensive input for subsequent analysis and decision-making. Then, the user's current communication task intention is analyzed and acquired according to the acquired operation information, audio information, and video information, so as to understand the actual demand of the user and enable the system to perceive the user's focus on different media streams in a specific situation. Subsequently, the acquired network state information, device performance information, and user communication task intention are taken as input to acquire parameter adjustment strategies for different media streams. Since the parameter adjustment strategies of the method are differentiated solutions that comprehensively consider the network, device, and user communication task intention, the method can ensure that the quality of media streams that are essential to complete the current task is prioritized when resources are limited. Finally, for each media stream, parameter adjustment is performed according to the acquired corresponding parameter adjustment strategy to apply differentiated strategies to actual media stream processing, thereby realizing fine-grained control of different media streams. The entire process forms a closed loop from perceiving user intention to generating differentiated strategies to executing adjustment, ensuring that the communication quality can better serve the dynamic task needs of the user.

[0033] As a preferred embodiment, taking the example of a user using a smartphone to remotely diagnose a device fault, the scheme of the application is implemented as follows: Step S1: the system obtains network state information of the smartphone (for example, Wi-Fi signal strength is medium, bandwidth is about 2 Mbps, packet loss rate is 1%, and delay is 50 ms), device performance information of the smartphone (for example, the processor model is mid-end, the screen resolution is 1080p, the camera supports 1080p@30fps, and the microphone is built-in), and user operation information in the communication application (for example, the user has performed multiple screen scaling and sliding operations, and has aimed the camera at a specific area of the device), voice information, and video information. Step S2: when the operation information is that there are multiple scaling and sliding operations, the audio information is that the user's voice contains a description of a specific component, and the video information is that the screen focus is concentrated on the device component, the screen has slight shaking but the target area is relatively stable, the system analyzes and judges that the user's communication task intention is “showing device details” according to the operation information, the audio information, and the video information. Step S3: the system determines the parameter adjustment strategy of different media streams according to the medium network bandwidth, the mid-end device performance, and the user's communication task intention of “showing device details”, specifically, the parameter adjustment strategy of each media stream is as follows: the uplink video stream (showing details) should give priority to guaranteeing the resolution and frame rate (for example, the resolution is 720p and the frame rate is 60fps); the downlink video stream (watching a remote expert) can appropriately reduce the resolution (for example, the resolution is 480p and the frame rate is 25fps); the uplink audio stream and the downlink audio stream maintain standard voice quality. Step S4: the system sets the resolution of the uplink video stream to 720p and the frame rate to 60fps according to the parameter adjustment strategy, selects appropriate encoding parameters, sets the resolution of the downlink video stream to 480p and the frame rate to 25fps according to the parameter adjustment strategy, and keeps the sampling rate and encoding parameters of the audio stream unchanged according to the parameter adjustment strategy. Through these adjustments, the system prioritizes the quality of the video details that the user wants to show under the limited network and device conditions, while maintaining the continuity of the overall communication.

[0034] Through the above scheme, the application can perform differential parameter adjustment on different media streams according to the dynamic changes of the user's communication task in combination with the network and device conditions, thereby solving the problem that the “one-size-fits-all” adjustment strategy in the prior art cannot meet the requirements of specific tasks, and effectively improving the effectiveness and user experience of audio and video communication in complex environments.

[0035] In some preferred embodiments, step S3 comprises:

[0036] S31, obtaining a first preliminary adjustment strategy corresponding to different media streams according to the network state information, the device performance information, and the user communication task intention;

[0037] S 32, obtaining noise parameter information according to the audio signal, the noise parameter information including noise type and noise intensity;

[0038] S33, correcting the first preliminary adjustment strategy corresponding to different media streams according to the noise parameter information, to obtain parameter adjustment strategies corresponding to different media streams.

[0039] The noise parameter information of the embodiment refers to data describing the characteristics of environmental noise in the current communication environment. The noise parameter information can be represented by noise type and noise intensity obtained by analyzing the audio signal. The noise type refers to the category of environmental noise, such as continuous background noise (e.g. fan sound, air conditioner sound) or burst noise (e.g. keyboard tapping sound, door opening sound). The embodiment can use audio signal processing techniques (e.g. spectrum analysis, feature extraction combined with a classifier) to identify the type of noise. The noise intensity refers to the loudness or energy level of environmental noise. The embodiment can use methods such as calculating the root mean square value, decibel value, or energy of a specific frequency band of the noise signal to obtain the noise intensity. The first preliminary adjustment strategy refers to a basic, further optimized parameter adjustment scheme determined for different media streams (e.g. uplink audio, downlink audio, uplink video, downlink video) after considering network status, device performance, and user communication task intent. The first preliminary adjustment strategy can include the recommended adjustment direction or numerical value of parameters such as code rate, resolution, frame rate, encoder parameters, packet loss retransmission strategy, and jitter buffer size. Correcting the first preliminary adjustment strategy corresponding to different media streams according to the noise parameter information refers to adjusting or modifying the initially determined parameter adjustment strategy (first preliminary adjustment strategy) according to additional environmental information (noise parameter information) to make it more suitable for the current actual environment. The embodiment can use rule-based adjustment, table lookup, or model-based prediction and optimization to achieve the correction of the first preliminary adjustment strategy corresponding to different media streams according to the noise parameter information.

[0040] The scheme optimizes the determination process of the parameter adjustment strategy by introducing the perception and utilization of environmental noise, so that the final adjustment strategy can better adapt to the actual communication environment, especially in the presence of noise. First, the first preliminary adjustment strategy corresponding to different media streams is obtained according to network state information, device performance information and user communication task intention. Then, the current audio environment quality is actively perceived and the environmental noise that may affect the communication quality is identified by acquiring noise parameter information from the audio signal, providing a detailed basis for subsequent strategy modification. Finally, the first preliminary adjustment strategy corresponding to different media streams is modified according to the obtained noise parameter information to obtain the final parameter adjustment strategy. This embodiment can make the final parameter adjustment strategy more adaptive to the current noise environment by modifying the preliminary strategy using noise parameter information, for example, if a specific type of noise or high-intensity noise is detected, the system will modify the first preliminary adjustment strategy corresponding to the audio stream (for example, increase the noise reduction processing strength, adjust the audio encoding parameters or increase the audio code rate to ensure the clarity and intelligibility of the voice); At the same time, it may also modify the first preliminary adjustment strategy corresponding to the video stream (for example, appropriately reduce the video quality to save bandwidth, and prioritize the stability of audio transmission). This noise information-based modification can make the parameter adjustment strategy no longer rely solely on network, device and task intention, but can dynamically respond to environmental changes, thereby providing a more optimized communication experience and better meeting the user's communication needs in a noisy environment. In addition, modifying the preliminary strategy of different media streams according to noise parameter information reflects the ability to take differentiated adjustments for different media streams, for example, noise mainly affects the audio stream, but its impact on the video stream may be reflected in bandwidth competition or encoding efficiency, so different correction measures need to be taken for different media streams according to noise information to achieve the overall optimal effect.

[0041] In an embodiment, obtaining the first preliminary adjustment strategy corresponding to different media streams according to the network state information, the device performance information and the user communication task intention can be achieved by a strategy generation module, which receives the network state, the device performance and the user task intention as inputs, and outputs a preliminary strategy set containing suggested parameters of each media stream. Obtaining the noise parameter information according to the audio signal can be achieved by an audio analysis module, which continuously listens to the audio signal collected by the local microphone, then performs spectral analysis and energy calculation on the signal, and identifies the noise type (such as office noise, street noise, wind noise) and calculates the noise intensity (such as in decibels) using a pre-trained classification model. Modifying the first preliminary adjustment strategy corresponding to different media streams according to the noise parameter information can be achieved by a strategy modification module, which receives the preliminary strategy set and the noise parameter information as inputs. For example, if the noise type is continuous office background noise and the intensity is high, the modification module can increase the noise reduction level of the uplink audio stream, and may fine-tune the parameters of the audio encoder to improve the speech coding efficiency; if the noise is a sudden keyboard click sound, the modification module can adjust the jitter buffer strategy of the audio to smooth the audio stream; if the noise intensity is extremely high, in order to guarantee the intelligibility of speech, the modification module can even suggest to appropriately reduce the code rate of the uplink video, and allocate the saved bandwidth to the uplink audio stream. This embodiment can make the audio and video parameter adjustment strategy more refined and environment adaptive by obtaining the noise parameter information from the audio signal and modifying the preliminary adjustment strategy based on the information, so as to optimize the parameters of the audio stream (such as enhancing noise reduction, adjusting coding) in the presence of environmental noise, while considering the indirect impact of noise on other media streams (such as video stream) (such as bandwidth competition), thereby effectively improving the clarity and intelligibility of speech in noisy environments, guaranteeing the effective transmission of key information, and improving the overall communication experience of users, better meeting the communication needs of users in different environments.

[0042] In some preferred embodiments, step S33 comprises:

[0043] S331, modifying the first preliminary adjustment strategy corresponding to different media streams according to the noise parameter information to obtain the second preliminary adjustment strategy corresponding to different media streams;

[0044] S332, obtaining network transmission data features;

[0045] S333, obtaining electromagnetic interference parameter information according to the network transmission data features;

[0046] S334, modifying the second preliminary adjustment strategy corresponding to different media streams according to the electromagnetic interference parameter information to obtain the parameter adjustment strategy corresponding to different media streams.

[0047] The network transmission data feature of this embodiment can substantially reflect the actual condition of the network transmission link, and preferably includes the loss pattern, retransmission condition and specific error indication of the data packet. The network transmission data feature can be obtained by monitoring the real-time data packet transmission of the network connection. The electromagnetic interference parameter information of this embodiment refers to data describing the type, intensity, frequency and other characteristics of electromagnetic interference in the current communication environment. This embodiment can obtain the electromagnetic interference parameter information by analyzing the network transmission data feature (such as the retransmission rate, delay jitter, bit error rate and other abnormal patterns of the data packet) to indirectly infer the existence and degree of electromagnetic interference. This embodiment can also obtain the electromagnetic interference parameter information by analyzing the correlation between the network transmission data feature and known electromagnetic interference patterns. For example, by analyzing the abnormal delay, burst loss or specific data damage pattern of the data packet based on a preset threshold, pattern matching algorithm or machine learning model, it can be inferred whether there is electromagnetic interference and its intensity and type to obtain the electromagnetic interference parameter information. The electromagnetic interference information of this embodiment can indicate the existence, intensity, duration or influence pattern of electromagnetic interference on specific data streams. The second preliminary adjustment strategy corresponding to different media streams is modified according to the electromagnetic interference parameter information, which means that the second preliminary adjustment strategy is adjusted or modified according to additional electromagnetic interference information (electromagnetic interference parameter information) to make it more suitable for the current actual environment. This embodiment can use rule-based adjustment, lookup table or model-based prediction and optimization to modify the second preliminary adjustment strategy corresponding to different media streams according to the electromagnetic interference parameter information. For example, a preset electromagnetic interference correction rule library is queried according to the electromagnetic interference parameter information to determine the correction amount of the second preliminary adjustment strategy, and then the second preliminary adjustment strategy is corrected based on the correction amount (for example, when the electromagnetic interference parameter information is a specific pattern of packet loss or check error, the forward error correction (FEC) is increased or the code rate is reduced to improve the reliability of data transmission).

[0048] Specifically, the scheme introduces the acquisition of network transmission data characteristics and secondary correction based on electromagnetic interference parameter information on the basis of correcting the first preliminary adjustment strategy corresponding to different media streams to obtain the second preliminary adjustment strategy according to noise parameter information. First, the network transmission data characteristics are acquired, which reflect the actual situation of data transmission in the network, for example, the loss mode of data packets. Then, the electromagnetic interference parameter information is analyzed and acquired according to the network transmission data characteristics. Electromagnetic interference may cause signal distortion or data transmission anomaly. This embodiment can indirectly or directly detect and quantify the influence of such interference by analyzing network transmission data characteristics. Finally, the electromagnetic interference parameter information obtained is used to further correct the second preliminary adjustment strategy that has been corrected by noise to obtain the final parameter adjustment strategy. This process is equivalent to superimposing the influence of electromagnetic interference on the second preliminary adjustment strategy that has considered the influence of noise. This step-by-step and multi-factor correction process can make the parameter adjustment strategy more finely adapt to complex actual communication environments, thereby ensuring the continuity of communication while better meeting the needs of user communication tasks and enabling the final parameter adjustment strategy to cope with both noise and electromagnetic interference, two complex environmental factors. This embodiment can make the parameter adjustment strategy more comprehensively consider complex interference environments by introducing the analysis of network transmission data characteristics and correction based on electromagnetic interference parameter information, so as to avoid the problem that a correction strategy considering only noise parameter information is insufficient to cope with complex communication environments, that is, the parameter adjustment strategy of this embodiment not only copes with noise but also copes with electromagnetic interference and other factors that may affect data transmission quality, thereby effectively improving the accuracy of parameter adjustment.

[0049] As a preferred embodiment, the scheme of the present application is implemented as follows: after obtaining the first preliminary adjustment strategy corresponding to different media streams, the system performs spectral analysis and pattern recognition on the collected audio signals to determine that the noise type of the current environment is machine roar and the noise intensity is medium, and then modifies the first preliminary adjustment strategy based on the noise type and noise intensity, for example, for the uplink audio stream, the gain and sampling rate are reduced; for the downlink audio stream, the noise reduction processing parameters are enhanced, to obtain the second preliminary adjustment strategy. Subsequently, the system monitors the network transmission, and when the average delay of the data packet suddenly increases and periodic packet loss occurs, the system inputs these network transmission data characteristics into the electromagnetic interference analysis module (which can analyze the strength and type of electromagnetic interference according to the input network transmission data characteristics) to obtain electromagnetic interference parameter information represented as "medium-strength radio frequency interference". Finally, the system further modifies the second preliminary adjustment strategy according to the electromagnetic interference parameter information, for example, for the uplink video stream, the code rate of the non-key area is appropriately reduced under the premise of ensuring the clarity of the key area to cope with possible bandwidth fluctuations and packet loss, and for the downlink video stream, its cache strategy is adjusted to adapt to the increase in delay to obtain a parameter adjustment strategy that takes into account the effects of noise and electromagnetic interference. This embodiment can make the parameter adjustment strategy more comprehensively consider the complex interference environment by step-by-step modification of the first preliminary adjustment strategy of the media stream based on noise parameter information and electromagnetic interference parameter information, thereby effectively improving the accuracy and robustness of the parameter adjustment strategy. Therefore, in a complex communication environment with noise and electromagnetic interference, this embodiment can generate a media stream parameter adjustment strategy that is more suitable for the current environment, thereby better maintaining or improving the quality of the media stream and meeting the needs of users in different communication tasks while ensuring communication continuity.

[0050] In some preferred embodiments, step S334 comprises:

[0051] A1, modifying the second preliminary adjustment strategy corresponding to different media streams according to the electromagnetic interference parameter information to obtain a third preliminary adjustment strategy corresponding to different media streams;

[0052] A2, obtaining role information of the user in the current conference;

[0053] A3, modifying the third preliminary adjustment strategy corresponding to different media streams according to the role information to obtain the parameter adjustment strategy.

[0054] The role information of the user in the current conference refers to the identity or responsibility (for example, a speaker, a listener, a host, or a screen sharer) of the user in the current communication session, and the embodiment can obtain the role information of the user in the current conference in a manner of obtaining from a conference server, analyzing according to a user behavior mode, or manually setting by the user. The parameter adjustment strategy of the embodiment refers to a final scheme for guiding the adjustment of the media stream parameters obtained by further correcting the third preliminary adjustment strategy according to the role information of the user in the current conference. The embodiment can implement the correction of the third preliminary adjustment strategy corresponding to different media streams according to the role information to obtain the parameter adjustment strategy in a manner of consulting a preset role-demand mapping table according to the user role information to obtain corresponding communication demands, and adjusting the third preliminary adjustment strategy according to the communication demands.

[0055] The method corrects the adjustment strategy of the media stream by combining the electromagnetic interference parameter information and the role information of the user in the current conference. First, the second preliminary adjustment strategy is corrected according to the electromagnetic interference parameter information to obtain an intermediate strategy with a certain robustness in the electromagnetic interference environment. Then, the role information of the user in the current conference is obtained, which reflects the core demands and priorities of the user in the current communication task. Finally, the intermediate strategy is corrected based on the role information of the user. This step-by-step correction method can make the final parameter adjustment strategy not only cope with the risk of communication quality decline caused by electromagnetic interference, but also fully consider the differentiated demands of different user roles. For example, for the speaker role, the system may prioritize ensuring the quality of the uplink audio and video stream under the premise of ensuring basic anti-interference capability, and maintain a high code rate and resolution as much as possible even under certain electromagnetic interference. For the listener role, the stability and clarity of the downlink audio stream may be prioritized. The embodiment can make the obtained strategy more fine and intelligent to adapt to complex communication scenarios by combining environmental factors (noise interference and electromagnetic interference) and user demand factors (role), so as to more effectively meet the communication quality requirements of the user in a specific task and achieve personalized and task-oriented adjustment closer to the actual demands of the user.

[0056] For example, in a remote technical support meeting, user A plays the role of "on-site operator" and needs to clearly show the device details through the uplink video stream; user B plays the role of "remote expert" and mainly relies on the downlink video stream for observation and the downlink audio stream for guidance. Assuming that there is moderate electromagnetic interference in the current environment. First, the system corrects the second preliminary adjustment strategy based on the network, device, and task intention according to the electromagnetic interference parameter information to obtain a third preliminary adjustment strategy, which may tend to generally reduce the code rate to enhance the anti-interference capability. Next, the system obtains the role information of user A as "on-site operator" and the role information of user B as "remote expert". Then, according to these role information, the third preliminary adjustment strategy is further corrected. For user A, although there is electromagnetic interference, the system will correct the strategy of the uplink video stream according to his "on-site operator" role, for example, on the basis of ensuring a certain anti-interference capability, appropriately increasing the minimum code rate of the uplink video stream or prioritizing the resolution of key areas to ensure the recognizability of device details. For user B, the system will correct the strategy of the downlink video stream and the downlink audio stream according to his "remote expert" role, for example, prioritizing the stability of the downlink audio stream and optimizing the strategy of the downlink video stream according to his observation needs.

[0057] In some preferred embodiments, step S2 comprises:

[0058] S21, obtaining an operation duration period according to the operation information;

[0059] S22, analyzing the continuity, stability, and repeatability of the operation information and the correlation with the device motion state to obtain operation characteristics;

[0060] S23, analyzing whether there is human voice in the audio information in the operation duration period, the intelligibility of the human voice, and the type and intensity of the environmental noise to obtain audio characteristics, and analyzing the picture stability, picture focus area, and picture content change of the video information in the operation duration period to obtain video characteristics;

[0061] S24, analyzing whether the operation information is an operation with clear intention according to the operation characteristics, video characteristics, and audio characteristics;

[0062] S25, when the operation information is an operation with clear intention, obtaining the user communication task intention according to the operation information, video characteristics, and audio characteristics.

[0063] According to the operation information, the operation duration period refers to determining the start time and end time of a user performing a certain operation behavior. This embodiment can monitor the time stamp of user input events (such as touch screen clicks, swipes, keyboard inputs, mouse operations, etc.), and use the record operation event occurrence and end time to define a continuous operation time window to obtain the operation duration period. Analyzing the continuity, stability, repeatability of the operation information and the correlation with the device motion state refers to quantitatively analyzing the performance of the obtained operation information in the duration period to obtain the operation characteristics. This embodiment can measure the stability of the operation information by calculating the average speed, acceleration and trajectory smoothness of the operation. This embodiment can measure the repeatability of the operation information by counting the frequency of the operation in a period of time. This embodiment can measure the user's operation mode by combining the data of the built-in sensors of the device (such as accelerometer, gyroscope) to determine whether the operation is synchronized or associated with the movement or posture change of the device. For example, a long and smooth swipe may indicate scrolling, a short and repeated click may indicate selection or confirmation, and an operation occurring simultaneously with rapid movement of the device may indicate that the user is operating while walking. Analyzing whether there is human voice, the intelligibility of human voice, and the type and intensity of environmental noise in the operation duration period refers to processing and analyzing the collected audio signals in the time window of the operation to obtain audio features. This embodiment can use a voice activity detection (VAD) algorithm to determine whether there is human voice. This embodiment can use a signal-to-noise ratio (S N R) or a specific voice quality evaluation index to measure the intelligibility of human voice. This embodiment can use an audio classification algorithm to identify the type of environmental noise (such as wind noise, mechanical noise, multiple people speaking, etc.) and calculate its energy or loudness to measure the intensity of environmental noise. For example, the presence of clear human voice may indicate that the user is speaking or listening to a voice, and high-intensity environmental noise may indicate that the user is in a noisy environment. Analyzing the picture stability, focal area and picture content change of the video information in the operation duration period refers to processing and analyzing the collected video frames in the time window of the operation to obtain video features. This embodiment can measure the picture stability by calculating the motion vector or image similarity between adjacent frames. This embodiment can use face detection, object detection or saliency region detection algorithms to identify the focal area in the picture. This embodiment measures the degree of change of picture content by using image difference calculation or feature point matching to measure the degree of change of picture content. For example, a stable picture with a focused area may indicate that the user is carefully observing an object, and a rapidly changing picture may indicate that the user is moving or switching perspectives.According to the operation feature, the video feature and the audio feature, analyzing whether the operation information is an operation with clear intention refers to comprehensively utilizing the quantitative features obtained in the previous steps to determine whether the current operation behavior is performed by the user actively and purposefully. The embodiment can use a rule-based method (for example, setting thresholds for duration, stability and repetition) or a machine learning model-based method (for example, training a classifier to distinguish between conscious operation and unconscious operation) to realize the analysis of whether the operation information is an operation with clear intention. For example, an operation with short duration, poor stability and high correlation with device motion can be determined as an accidental operation without clear intention. When the operation information is an operation with clear intention, according to the operation information, the video feature and the audio feature, analyzing to obtain the user communication task intention refers to determining the specific task that the user wants to complete in the communication application in combination with the type of operation (for example, zoom gesture, sliding, clicking specific button) and the audio and video features at the same period on the premise that the operation is determined to have clear intention. The embodiment can use a pre-set intention recognition rule library (for example, zoom gesture + focus of the picture + human voice explanation = detail display intention) or train a multi-modal intention recognition model to realize the analysis to obtain the user communication task intention. For example, when the user performs a zoom gesture in a video call, the focus area of the picture is concentrated on an object, and there is human voice in the audio to describe the object, it can be inferred that the task intention of the user is “to display and explain the details of an object to the far end”. Since the embodiment only analyzes to obtain the user communication task intention according to the operation information, the video feature and the audio feature when the operation information is determined to be an operation with clear intention, the embodiment can improve the accuracy and reliability of the user communication task intention, thereby providing a reliable basis for subsequent adaptive parameter adjustment according to the user intention.

[0064] In some preferred embodiments, S25 comprises:

[0065] S251, when the operation information is an operation with clear intention, obtaining noise parameter information according to the audio signal;

[0066] S252, obtaining network transmission data features, and obtaining electromagnetic interference parameter information according to the network transmission data features;

[0067] S253, determining a task intention analysis rule according to the noise parameter information and the electromagnetic interference parameter information;

[0068] S254, using the task intention analysis rule to analyze to obtain the user communication task intention according to the operation information, the video feature and the audio feature.

[0069] The task intent analysis rule of this embodiment refers to a logic or model used to guide how to combine operation information, video features and audio features to determine the user communication task intent. The task intent analysis rule can be a rule set constructed based on expert knowledge, a decision tree model or a classification model trained through machine learning.

[0070] This embodiment is equivalent to determining that the user operation has a clear intent, and further considering the influence of environmental factors on the accuracy of intent analysis. Specifically, after determining that the operation information has a clear intent, the system does not directly perform intent analysis, but first analyzes the current audio signal to obtain noise parameter information and obtains network transmission data features and electromagnetic interference parameter information based on which to obtain the potential influence of environmental noise on audio information quality and the potential influence of electromagnetic interference on device perception and data transmission. Subsequently, the task intent analysis rule used for subsequent intent analysis is dynamically determined or adjusted according to the noise parameter information and the electromagnetic interference parameter information, which means that the strategy of intent analysis is no longer fixed, but can be adaptively adjusted according to the actual environment in which it is currently located. For example, when high-intensity noise is detected, the system adjusts the task intent analysis rule (for example, reduces the weight of audio features in determining user intent or preferentially relies on video features and operation information that are not sensitive to noise); when electromagnetic interference is detected, the system adjusts the task intent analysis rule (for example, checks or reduces the weight of data sources that may be affected by interference, such as certain sensor data, or uses an analysis model that is more robust to data incompleteness). Finally, the task intent analysis rule adjusted according to the environmental factors is used to combine the current operation information, video features and audio features to perform the final user communication task intent analysis. In this way, even in the presence of environmental noise or electromagnetic interference, this embodiment can more accurately interpret the user's real communication task intent, i.e. this embodiment can significantly improve the accuracy and robustness of user communication task intent analysis, to more reliably identify the user's communication needs and provide a more solid foundation for subsequent media stream parameter adjustment.

[0071] For example, in a specific implementation scenario, a user can be in a strong mechanical noise environment for remote video guidance. The user tries to show a certain detail of the device to the remote expert by continuously zooming and pointing on the screen. At this time, the system first determines that the user's zooming and pointing operations have a clear intention. Then, by analyzing the audio signal, the system detects high-intensity mechanical noise and obtains the corresponding noise parameter information. At the same time, by analyzing the network transmission data characteristics, the system can find that there is a certain fluctuation in data transmission, and accordingly obtain the electromagnetic interference parameter information. Based on these noise parameters and electromagnetic interference parameters, the system dynamically adjusts the task intention analysis rules, for example, reduces the weight of the intelligibility of human voice in the audio on the judgment of whether the user is "explaining", and increases the weight of the stability of the focus area of the video picture and the user's operation (such as zooming and pointing) on the judgment of whether the user is "showing details". Finally, the system uses the adjusted rules to analyze and determine that the user's communication task intention is "showing device details" by combining the operation information of the user's continuous zooming and pointing, the video features of the video picture focusing on the device details, and the audio features affected by the noise.

[0072] In some preferred embodiments, the media stream is an uplink audio stream, a downlink audio stream, an uplink video stream, or a downlink video stream. This embodiment is equivalent to realizing differentiated parameter adjustment of different media streams by explicitly specifying the specific type of media stream, so as to avoid the problem of quality loss caused by uniform processing of all media streams in a complex environment and dynamic task. Therefore, this embodiment can make the system prioritize the quality of key media streams according to the user's communication task intention, thereby effectively improving the adaptability and effectiveness of communication.

[0073] In some preferred embodiments, step S4 comprises:

[0074] S41, for the uplink audio stream and the downlink audio stream, adjusting the parameters of the uplink audio stream or the downlink audio stream according to the corresponding parameter adjustment strategy;

[0075] S42, obtaining the key area and the non-key area according to the video information analysis;

[0076] S43, for the uplink video stream and the downlink video stream, correcting the parameter adjustment strategy corresponding to the key area and the parameter adjustment strategy corresponding to the non-key area according to the proportion of the key area to the non-key area, then adjusting the parameters of the area corresponding to the key area in the uplink video stream or the downlink video stream according to the parameter adjustment strategy corresponding to the key area, and adjusting the parameters of the area corresponding to the non-key area in the uplink video stream or the downlink video stream according to the parameter adjustment strategy corresponding to the non-key area.

[0077] According to the video information analysis to obtain the key area and the non-key area refers to the process of understanding the video content to identify the area (key area) important to the user's current communication task and the relatively unimportant area (non-key area) in the video picture. This embodiment can specifically use various technologies to realize the analysis of the video information to obtain the key area and the non-key area, for example: based on image processing and computer vision technology (face detection, object recognition, motion area detection, focus area analysis, saliency detection, etc. Algorithm) to identify specific objects, active areas or user attention areas in the video picture, and take the area as the key area, and then take the area in the video picture except the key area as the non-key area. Step S43 corresponds to differentiating adjustment of the video stream parameters based on the analysis results of the key area and the non-key area. First, according to the proportion of the identified key area and non-key area in the picture, the original parameter adjustment strategy corresponding to the key area and the non-key area is corrected, for example, if the key area proportion is large, the strategy can be adjusted appropriately to ensure the overall key information quality; if the key area proportion is small, more resources can be concentrated in the key area. Specifically, this embodiment can use the method of calculating the weight according to the proportion of the key area and the non-key area, and then applying the weight to the parameter adjustment strategy to correct the parameter adjustment strategy corresponding to the key area and the parameter adjustment strategy corresponding to the non-key area based on the proportion of the key area and the non-key area. Then, according to the parameter adjustment strategy corresponding to the corrected key area, the part of the video stream belonging to the key area is adjusted to ensure the quality of the key area (for example, allocate higher code rate, use higher resolution or more detailed encoding parameters for the key area), and at the same time, according to the parameter adjustment strategy corresponding to the corrected non-key area, the part of the video stream belonging to the non-key area is adjusted to save resources (for example, allocate lower code rate, reduce resolution, apply blur processing or use coarser encoding parameters for the non-key area). This embodiment can distinguish the key area and the non-key area according to the video information, and apply different parameter adjustment strategies to the key area and the non-key area, so that the system can preferentially guarantee the video quality of the key area to ensure that the details of the user's attention are clear and visible, thereby better meeting the user's communication task requirements for the clarity of a specific area, while ensuring the quality of key information transmission, and improving resource utilization efficiency by reducing the quality of the non-key area to save bandwidth and computing resources.

[0078] In some preferred embodiments, the network status information includes network bandwidth, packet loss rate and network delay. When obtaining the parameter adjustment strategy corresponding to different media streams according to the network status information, the device performance information and the user communication task intention, the system can use accurate network bandwidth data to evaluate the transmission capacity of the current network, so as to reasonably allocate the code rate and resolution, and avoid network congestion or bandwidth waste. At the same time, the use of packet loss rate data can evaluate the stability of the network, guide the system to take forward error correction or retransmission strategy, and reduce the influence of data loss on the quality of media stream. In addition, the use of network delay data can evaluate the real-time performance of the network, guide the system to adjust the buffer size and synchronization strategy of the media stream, and improve the fluency of interaction.

[0079] As can be seen from the above, the voice and video integrated communication method across devices and platforms provided by the present application can perceive the current communication task intention of the user by obtaining and analyzing the operation information, audio information and video information of the user in the communication process, and then dynamically determine the differentiated parameter adjustment strategy for different media streams by combining the communication task intention with the traditional network status information and device performance information. Finally, the parameters of different media streams are adjusted based on the parameter adjustment strategy. Since the adaptive adjustment strategy of the present application can adapt to the dynamic changes of device performance, network status parameters and communication tasks, that is, when the network quality decreases and the device is limited, the present application will not uniformly and symmetrically reduce the quality of all media streams, that is, the present application will not adopt the "one-size-fits-all" strategy when the network quality decreases and the device is limited. Therefore, the present application can effectively solve the problem that the adaptive adjustment strategy based on device performance and network status parameters cannot adapt to the dynamic changes of communication tasks, so that the audio and video quality cannot meet the needs of the communication task when the communication task changes, thereby effectively improving the effectiveness of communication.

[0080] In a second aspect, as shown in Figure 2 The present application further provides a voice and video integrated communication system across devices and platforms, which comprises:

[0081] An information acquisition module 1 is configured to acquire network status information, device performance information, and operation information, audio information and video information of a user in a communication application;

[0082] A task intention analysis module 2 is configured to analyze and acquire the communication task intention of the user according to the operation information, audio information and video information;

[0083] A parameter adjustment strategy determination module 3 is configured to obtain the parameter adjustment strategy corresponding to different media streams according to the network status information and the communication task intention of the user;

[0084] A parameter adjustment module 4 is configured to adjust the parameters of each media stream according to the corresponding parameter adjustment strategy.

[0085] The voice and video integrated communication system across devices and platforms provided in the application comprises an information acquisition module 1, a task intention analysis module 2, a parameter adjustment strategy determination module 3 and a parameter adjustment module 4. The voice and video integrated communication system across devices and platforms provided in the embodiment is used to execute the steps in the voice and video integrated communication method across devices and platforms provided in the first aspect. The principle of the voice and video integrated communication system across devices and platforms provided in the embodiment is the same as that of the voice and video integrated communication method across devices and platforms provided in the first aspect, which will not be discussed in detail here.

[0086] As can be seen from the above, the voice and video integrated communication method and system across devices and platforms provided in the application can perceive the current communication task intention of a user by acquiring and analyzing the operation information, audio information and video information of the user in the communication process, and then dynamically determine the differentiated parameter adjustment strategy for different media streams by combining the communication task intention with the traditional network state information and device performance information, and finally adjust the parameters of different media streams based on the parameter adjustment strategy. Since the adaptive adjustment strategy of the application can adapt to the dynamic changes of device performance, network state parameters and communication tasks, that is, when the network quality decreases and the device is limited, the application will not uniformly and symmetrically reduce the quality of all media streams, that is, the application will not adopt the "one-size-fits-all" strategy when the network quality decreases and the device is limited. Therefore, the application can effectively solve the problem that the adaptive adjustment strategy based on device performance and network state parameters cannot adapt to the dynamic changes of communication tasks, so that the quality of audio and video cannot meet the requirements of the communication task when the communication task changes, thereby effectively improving the effectiveness of communication.

[0087] In the embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the above units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another robot, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0088] In addition, each functional module in each embodiment of the application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0089] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0090] The above merely provides an example of the present application, but does not serve to limit the protection scope of the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-device and platform integrated voice and video communication method, characterized in that, The cross-device and cross-platform integrated voice and video communication method includes the following steps: S1. Obtain network status information, device performance information, and user operation information, audio information, and video information in communication applications; S2. Analyze and obtain the user's communication task intent based on the operation information, audio information, and video information; S3. Obtain parameter adjustment strategies corresponding to different media streams based on the network status information, the device performance information, and the user communication task intent; S4. For each media stream, adjust the parameters of the media stream according to the corresponding parameter adjustment strategy.

2. The cross-device and cross-platform integrated voice and video communication method according to claim 1, characterized in that, Step S3 includes: S31. Obtain a first preliminary adjustment strategy corresponding to different media streams based on the network status information, the device performance information, and the user communication task intent; S32. Obtain noise parameter information based on the audio signal, wherein the noise parameter information includes noise type and noise intensity; S33. Based on the noise parameter information, the first preliminary adjustment strategy corresponding to different media streams is modified to obtain the parameter adjustment strategy corresponding to different media streams.

3. The cross-device and cross-platform integrated voice and video communication method according to claim 2, characterized in that, Step S33 includes: S331. Based on the noise parameter information, the first preliminary adjustment strategy corresponding to different media streams is modified to obtain the second preliminary adjustment strategy corresponding to different media streams. S332. Obtain the characteristics of network transmission data; S333. Obtain electromagnetic interference parameter information based on the characteristics of the network transmission data; S334. Based on the electromagnetic interference parameter information, the second preliminary adjustment strategy corresponding to different media streams is modified to obtain the parameter adjustment strategy corresponding to different media streams.

4. The cross-device and cross-platform integrated voice and video communication method according to claim 3, characterized in that, Step S334 includes: A1. Based on the electromagnetic interference parameter information, the second preliminary adjustment strategy corresponding to different media streams is modified to obtain the third preliminary adjustment strategy corresponding to different media streams. A2. Obtain the user's role information in the current meeting; A3. Based on the aforementioned role information, the third preliminary adjustment strategy corresponding to different media streams is modified to obtain the parameter adjustment strategy.

5. The cross-device and cross-platform integrated voice and video communication method according to claim 1, characterized in that, Step S2 includes: S21. Obtain the operation duration segment based on the operation information; S22. Analyze the continuity, stability, and repeatability of the operation information, as well as its correlation with the device's motion state, to obtain operation characteristics; S23. Analyze the audio information within the operation duration period to determine whether there is a human voice, the clarity of the human voice, and the type and intensity of environmental noise, in order to obtain audio features. Also analyze the video information within the operation duration period to determine the image stability, image focus area, and image content changes, in order to obtain video features. S24. Analyze whether the operation information is an operation with a clear intent based on the operation features, the video features, and the audio features; S25. When the operation information is an operation with a clear intent, the user's communication task intent is obtained by analyzing the operation information, the video features, and the audio features.

6. The cross-device and cross-platform integrated voice and video communication method according to claim 5, characterized in that, S25 includes: S251. When the operation information is an operation with a clear intention, noise parameter information is obtained based on the audio signal; S252. Obtain network transmission data characteristics, and obtain electromagnetic interference parameter information based on the network transmission data characteristics; S253. Determine the task intent analysis rules based on the noise parameter information and the electromagnetic interference parameter information; S254. Using the task intent analysis rules, the user's communication task intent is obtained by analyzing the operation information, the video features, and the audio features.

7. The cross-device and cross-platform integrated voice and video communication method according to claim 1, characterized in that, The media stream is an upstream audio stream, a downstream audio stream, an upstream video stream, or a downstream video stream.

8. The cross-device and cross-platform integrated voice and video communication method according to claim 7, characterized in that, Step S4 includes: S41. For the uplink audio stream and the downlink audio stream, adjust the parameters of the uplink audio stream or the downlink audio stream according to the corresponding parameter adjustment strategy; S42. Analyze and obtain key and non-key areas based on the video information; S43. For the uplink video stream and the downlink video stream, modify the parameter adjustment strategy corresponding to the key region and the parameter adjustment strategy corresponding to the non-key region according to the ratio of the key region to the non-key region. Then, adjust the parameters of the region corresponding to the key region in the uplink video stream or the downlink video stream according to the parameter adjustment strategy corresponding to the key region, and adjust the parameters of the region corresponding to the non-key region in the uplink video stream or the downlink video stream according to the parameter adjustment strategy corresponding to the non-key region.

9. The cross-device and cross-platform integrated voice and video communication method according to claim 1, characterized in that, The network status information includes network bandwidth, packet loss rate, and network latency.

10. A cross-device and cross-platform integrated voice and video communication system, characterized in that, The cross-device and cross-platform integrated voice and video communication system includes: The information acquisition module is used to acquire network status information, device performance information, and user operation information, audio information, and video information in communication applications. The task intent analysis module is used to analyze and obtain the user's communication task intent based on the operation information, audio information, and video information. The parameter adjustment strategy determination module is used to obtain the parameter adjustment strategy corresponding to different media streams based on the network status information and the user's communication task intent. The parameter adjustment module is used to adjust the parameters of each media stream according to the corresponding parameter adjustment strategy.

Images