Audio and video data transmission method and system, electronic device, and storage medium

Abstract

The application provides an audio and video data transmission method and system, an electronic device and a storage medium, and belongs to the field of video data transmission. The method comprises the following steps: obtaining parameter information corresponding to each target device, wherein the target devices are devices to be subjected to audio and video transmission; the parameter information of one target device comprises a plurality of parameters and respective corresponding parameter values; the plurality of parameters comprise performance parameters, network parameters and environmental parameters; for each target device, a comprehensive label value of the target device is calculated according to the parameter values corresponding to the target device and the weights corresponding to the parameter values; the audio and video transmission mode corresponding to each target device is determined according to the comprehensive label value corresponding to each target device, so that each target device performs data transmission based on the audio and video transmission mode corresponding to each target device; and the audio and video transmission mode comprises a first transmission mode, a second transmission mode or a third transmission mode.

Description

Technical Field

[0001] This application belongs to the field of data transmission technology, and more specifically, relates to an audio and video data transmission method and system, electronic device, and storage medium. Background Technology

[0002] Multi-person video conferencing, as a core tool for remote collaboration, is widely used in various fields such as business offices, education and training, and medical consultations. However, due to differences in hardware performance and network environments, transmitting audio and video data via multi-person video conferencing often suffers from poor device compatibility and adaptability, and the stability of audio and video data transmission needs improvement. Summary of the Invention

[0003] The purpose of this application is to provide an audio and video data transmission method and system, electronic device, and storage medium, so as to match the most suitable audio and video data transmission method for devices with different parameters.

[0004] A first aspect of this application provides an audio / video data transmission method, including:

[0005] Obtain the parameter information corresponding to each target device. Each target device is a device to be used for audio and video transmission. The parameter information of a target device includes multiple parameters and their corresponding parameter values. The multiple parameters include performance parameters, network parameters, and environmental parameters.

[0006] For each target device, a comprehensive tag value is calculated based on the parameter values ​​and their respective weights. The comprehensive tag value represents the overall adaptability of the target device in audio and video data transmission, and the comprehensive tag value is positively correlated with the adaptability.

[0007] The corresponding audio and video transmission mode is determined based on the comprehensive tag value of each target device, so that each target device can transmit data based on its corresponding audio and video transmission mode. The audio and video transmission mode includes a first transmission mode, a second transmission mode, or a third transmission mode.

[0008] The first and second transmission modes can transmit audio and video data to the device. The resolution of the video data transmitted in the first transmission mode is higher than the preset resolution, the resolution of the video data transmitted in the second transmission mode is no higher than the preset resolution, and the third transmission mode can only transmit audio data to the device.

[0009] A second aspect of this application provides an audio and video data transmission system, including:

[0010] The device information acquisition module is used to acquire the parameter information corresponding to each target device. Each target device is a device to be transmitted audio and video. The parameter information of a target device includes multiple parameters and their corresponding parameter values. The multiple parameters include performance parameters, network parameters and environmental parameters.

[0011] The comprehensive tag value acquisition module is used to calculate the comprehensive tag value of each target device based on the parameter values ​​and weights corresponding to each parameter value. The comprehensive tag value represents the adaptation performance of the target device, and the comprehensive tag value is positively correlated with the adaptation capability.

[0012] The transmission mode determination module is used to determine the corresponding audio and video transmission mode for each target device based on the comprehensive tag value, so that each target device can transmit data based on its corresponding audio and video transmission mode. The audio and video transmission modes include a first transmission mode, a second transmission mode, or a third transmission mode. The first and second transmission modes can transmit both audio and video data to the device. The resolution of the video data transmitted in the first transmission mode is higher than the preset resolution, the resolution of the video data transmitted in the second transmission mode is no higher than the preset resolution, and the third transmission mode can only transmit audio data to the device.

[0013] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described audio and video data transmission method.

[0014] A fourth aspect of this application provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described audio and video data transmission method.

[0015] The beneficial effects of the audio and video data transmission method and system, electronic device, and storage medium provided in this application are as follows: This application obtains parameter information of each target device, determines the comprehensive tag value of each target device based on the parameter value and its corresponding weight, and determines the transmission parameters of each target device based on the comprehensive tag value. This allows for a comprehensive and accurate evaluation of the compatibility and overall adaptability of each target device during audio and video data transmission. Furthermore, it determines the optimal audio and video transmission mode for each target device based on the comprehensive tag value, ensuring that each target device can transmit data using the most suitable transmission method, thus avoiding resource waste. This embodiment comprehensively considers the performance parameters, network parameters, and environmental parameters of the target device to select a suitable transmission mode, dynamically adjusting the transmission parameters according to the actual situation of the target device, thus ensuring the stability of data transmission under different network environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating an audio / video data transmission method provided in an embodiment of this application;

[0018] Figure 2 This is a structural block diagram of an audio and video data transmission system provided in an embodiment of this application;

[0019] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0022] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an audio / video data transmission method according to an embodiment of this application. The method can be executed by a server. In this embodiment, the audio / video data of each target device can first be sent to the server, and then the server forwards it to other target devices. The method may include:

[0023] S101: Obtain the parameter information corresponding to each target device. Each target device is a device to be used for audio and video transmission. The parameter information of a target device includes multiple parameters and their corresponding parameter values. The multiple parameters include performance parameters, network parameters and environmental parameters.

[0024] In this embodiment, with the help of an automatic device detection protocol (e.g., signaling interaction via Session Initiation Protocol or Real-Time Streaming Protocol), each target device actively reports its corresponding parameter information. The performance parameters of a device include its encoding and decoding capabilities, peripheral performance, basic device attributes, and load capacity; the network parameters of a device include uplink bandwidth parameters, connectivity attributes, real-time transmission performance, and network stability; and the environmental parameters of a device include spatial scene attributes and environmental interference factors.

[0025] For example, a technology company's marketing department conducts a remote demonstration of a new product to its sales department and clients, involving three types of target devices: the marketing department's live streaming equipment, devices connected to the sales department, and the client's home desktop computer. Parameter information for each type of target device is obtained. The performance parameters of the marketing department's live streaming equipment are actively queried through the device control protocol. These parameters include the camera's resolution and optical zoom, the microphone's sampling rate, signal-to-noise ratio, and pickup distance, and the console's maximum encoding rate. Network parameters include uplink bandwidth, jitter, and latency between the live streaming equipment connected to the sales department and the client's home desktop computer. The device connected to the sales department is an office laptop. Its performance parameters include the integrated camera's resolution, processor model, and graphics card type. Network parameters include real-time bandwidth, packet loss rate, and network type. Environmental parameters include lighting conditions and noise levels. The performance parameters of the client's home desktop computer include the external camera's resolution, headphone / microphone sampling rate, and processor information. Network parameters include bandwidth fluctuations, network latency, and network connection type. Environmental parameters include the noise level of the home environment and power stability. Intelligent adaptation is performed based on the three types of target devices collected, enabling seamless interaction between departments and allowing customers to view product demonstration functions clearly and smoothly.

[0026] S102: For each target device, calculate the comprehensive tag value of the target device based on the parameter values ​​and their respective weights. The comprehensive tag value represents the overall adaptability of the target device in audio and video data transmission, and the comprehensive tag value is positively correlated with the adaptability.

[0027] In this embodiment, weights are assigned to the parameter values ​​corresponding to each parameter of each target device. Performance parameters include encoding / decoding capability and load capacity. The encoding / decoding capability and load capacity are weighted to determine the performance parameter values ​​of the target device. Network parameters include bandwidth parameters and network stability. The bandwidth parameter, packet loss rate, and round-trip latency are weighted to determine the network parameter values ​​of the target device. Environmental parameters include environmental interference information. The environmental interference information is weighted to determine the environmental parameter values.

[0028] Based on the performance, network, and environmental parameters of each target device, a comprehensive label value is calculated for each target device according to the weights corresponding to each parameter value. This comprehensive label value characterizes the overall adaptability of the target device during audio and video data transmission. In this embodiment, the comprehensive label value is positively correlated with adaptability. For example, a comprehensive label value in the range of 0.7-1 indicates the highest overall adaptability of the target device in audio and video data transmission; a comprehensive label value in the range of 0.5-0.7 indicates the next highest; and a comprehensive label value in the range of 0-0.5 indicates the lowest. Based on the comprehensive label value of each target device, the optimal audio and video transmission mode is provided for each target device, achieving precise resource allocation and dynamic scene adaptation.

[0029] This embodiment also includes normalizing the parameter values ​​corresponding to the target device to obtain their respective normalization results; wherein, calculating the comprehensive label value of the target device based on each parameter value and its corresponding weight includes:

[0030] The comprehensive label value of the target device is determined by each normalization result and its corresponding weight.

[0031] In this embodiment, performance parameters, network parameters, and environmental parameters of each target device are acquired. The parameter values ​​for each parameter are then normalized to obtain normalized results, eliminating differences caused by different devices, data formats, and magnitudes. Weights are assigned to each normalized result to obtain the normalized weights for each parameter value. Finally, the comprehensive label value of the target device is determined based on each normalized result and its corresponding weight.

[0032] For example, the parameter values ​​of the target device are normalized to obtain the normalized results of each parameter. If the normalized encoding / decoding capability is 0.7 and the load capability is 0.5, and the weight of the encoding / decoding capability is set to 60% and the weight of the load capability is set to 40%, then the performance parameter value of the target device is 0.62. If the normalized bandwidth parameter is 0.9 and the network stability is 0.8, and the weight of the bandwidth parameter is set to 40% and the weight of the network stability is set to 60%, then the network parameter value of the target device is 0.84. If the normalized environmental interference information is 0.4, and the weight of the environmental interference information is set to 100%, then the environmental parameter value of the target device is 0.4.

[0033] The weights for performance parameters are set to 30%, network parameters to 60%, and environmental parameters to 10%. Based on the target device's parameter values ​​and their corresponding weights, the overall label value is calculated as 0.62 × 30% + 0.84 × 60% + 0.4 × 10% = 0.73. Based on this overall label value, the most suitable audio and video transmission mode is matched to the target device.

[0034] S103: Determine the corresponding audio and video transmission mode for each target device based on the comprehensive tag value, so that each target device can transmit data based on its corresponding audio and video transmission mode. The audio and video transmission mode includes a first transmission mode, a second transmission mode, or a third transmission mode.

[0035] The first and second transmission modes can transmit audio and video data to the device. The resolution of the video data transmitted in the first transmission mode is higher than the preset resolution, the resolution of the video data transmitted in the second transmission mode is no higher than the preset resolution, and the third transmission mode can only transmit audio data to the device.

[0036] In this embodiment, the most suitable audio and video data transmission mode for the target device is selected based on the target device's comprehensive tag value. The comprehensive tag value of the target device is determined according to parameters such as its encoding / decoding capabilities, CPU load, uplink bandwidth, and network stability, as well as the weights corresponding to each parameter value. Based on the comprehensive tag value, each target device selects the most suitable audio and video transmission mode for data transmission. The correspondence between the comprehensive tag value and the audio and video transmission mode in this embodiment is as follows: if the comprehensive tag value of the target device is in the range of 0.7-1, the target device transmits data through the first transmission mode, transmitting audio data and video data with a resolution higher than a preset resolution; if the comprehensive tag value of the target device is in the range of 0.5-0.7, the target device transmits data through the second transmission mode, transmitting audio data and video data with a resolution lower than a preset resolution; if the comprehensive tag value of the target device is in the range of 0-0.5, the target device transmits data through the third transmission mode, transmitting only audio data.

[0037] As can be seen from the above, this embodiment acquires the parameter information corresponding to each target device, evaluates the parameter information of each target device based on the parameter values ​​corresponding to the parameter information, thereby obtaining the comprehensive tag value corresponding to each target device. Based on the comprehensive tag value, the optimal audio and video transmission mode is matched for the target device, enabling each target device to transmit data according to the corresponding audio and video transmission mode. This embodiment forms a complete audio and video transmission optimization logic, comprehensively considering the performance parameters, network parameters, and environmental parameters of the target devices, as well as the parameter values ​​corresponding to each parameter, to accurately match the audio and video transmission mode for each target device, improve the network adaptability and compatibility of each device, and ensure the stability and smoothness of audio and video data transmission.

[0038] In one embodiment of this application, the comprehensive tag value of the target device is calculated based on the parameter values ​​corresponding to the target device and the weights corresponding to each parameter value, including:

[0039] The fault type of the parameter information is determined based on each parameter value. Fault types include network fluctuations, network outages, and sudden drops in device performance.

[0040] Set the weight of each parameter according to the fault type;

[0041] The comprehensive label value of the target device is calculated based on each parameter value and its corresponding weight.

[0042] In this embodiment, the parameter values ​​corresponding to the acquired parameter information of each target device are compared with preset threshold ranges corresponding to each parameter value. Based on the comparison results, the corresponding fault type of the current target device is determined. These fault types include network fluctuations, network interruptions, and sudden drops in device performance. Based on the determined fault type, the weights of each parameter are reallocated, and the comprehensive label value of the target device is determined based on the weight of each parameter value. This embodiment determines the comprehensive label value of the target device by adjusting the weights of its various parameter values, and then determines the audio and video data transmission mode of the target device based on the comprehensive label value, achieving optimal adaptation of the target device's audio and video transmission mode.

[0043] For example, if the CPU utilization of a target device suddenly increases from 50% to 80%, the weights of load capacity are reallocated, while interference from encoding / decoding capabilities and environmental capabilities is reduced. For instance, if the weight of encoding / decoding capability is set to 40%, and the weight of load capacity to 60%, the target device's performance parameter value is 0.58. If the weight of bandwidth parameter is set to 40%, and the weight of network stability to 60%, the target device's network parameter value is 0.84. If the weight of environmental interference information is set to 100%, the target device's environmental parameter value is 0.4. Alternatively, if the weights of the target device's performance parameter value are set to 40%, the network parameter value to 45%, and the environmental parameter value to 15%, the target device's overall label value is 0.67. At this point, the target device switches to the second transmission mode for audio and video data transmission based on the updated overall label value.

[0044] In one embodiment of this application, if the fault type of the target device is network fluctuation, the audio and video data transmission method further includes:

[0045] Based on the parameter values ​​corresponding to the network parameters, the network fluctuation level corresponding to the current network fluctuation is determined. The network fluctuation level is either Level 1, Level 2, or Level 3, with the fluctuation intensity corresponding to Level 1, Level 2, and Level 3 increasing sequentially.

[0046] The weights for each parameter are set according to the fault type, including:

[0047] If the network fluctuation level is Level 1, then the weights of each parameter value are set according to the first allocation mode.

[0048] If the network fluctuation level is level two, then the weights of each parameter value are set according to the second allocation mode.

[0049] If the network fluctuation level is level three, then the weights of each parameter value are set according to the third allocation mode.

[0050] Among them, the network parameter weights corresponding to the first allocation mode, the second allocation mode, and the third allocation mode decrease sequentially.

[0051] In this embodiment, the factor affecting network fluctuation is bandwidth jitter. Based on the parameter values ​​corresponding to the bandwidth parameters of the target device, the fluctuation level corresponding to the current network fluctuation is determined. In this embodiment, if the bandwidth fluctuation is less than or equal to 20%, the network fluctuation level is Level 1; if the bandwidth fluctuation is between 20% and 50%, the network fluctuation level is Level 2; and if the bandwidth fluctuation is greater than 50%, the network fluctuation level is Level 3. Weights are assigned to each parameter value according to the allocation mode corresponding to the network fluctuation level. The allocation mode includes a first allocation mode, a second allocation mode, or a third allocation mode, with the network parameter weights decreasing sequentially for the first, second, and third allocation modes. In this embodiment, the specific correspondence between network fluctuation level and allocation mode is as follows: If the network fluctuation level is Level 1, then according to the first allocation mode, each parameter value is assigned a corresponding weight. The first allocation mode refers to a bandwidth parameter weight of 35% and a network stability weight of 65%. Based on the first allocation mode, the network parameter value is determined to be 0.835. Under Level 1 fluctuation, the overall network performance is reliable, and the weight of network stability is appropriately increased to ensure the real-time performance of audio and video transmission. If the network fluctuation level is Level 2, then according to the second allocation mode, each parameter value is assigned a corresponding weight. The second allocation mode refers to a bandwidth parameter weight of 50% and a network stability weight of 65%. The weight of bandwidth is 50%, and the network parameter value is determined to be 0.85 based on the second allocation mode. Under the second level of fluctuation, the packet loss rate or latency may fluctuate to some extent. By increasing the weight of the bandwidth parameter, we can avoid blurry images or disconnections caused by fluctuations. If the network fluctuation is at the third level, the weight of each parameter value is set according to the third allocation mode. The third allocation mode refers to the bandwidth parameter having a weight of 70% and the network stability having a weight of 30%. Based on the third allocation mode, the network parameter value is determined to be 0.87. Under the third level of fluctuation, the network failure rate and latency may seriously exceed the standard. It is necessary to prioritize ensuring sufficient bandwidth to complete the transmission of audio and video data and avoid disconnections.

[0052] In one embodiment of this application, if the fault type of the target device is a network interruption, the audio and video data transmission method further includes:

[0053] Determine the network outage duration for the target device;

[0054] If the network interruption time of the target device exceeds the preset interruption time, the network interruption information of the target device is obtained. The network interruption information includes: audio and video data transmitted to the target device by other target devices during the network interruption time.

[0055] If the network connection to the target device is detected to have been restored, a network interruption message will be sent to the target device.

[0056] In this embodiment, the network interruption time of the target device is determined. If the network interruption time of the target device exceeds a preset interruption time, it is determined that the target device has experienced a network interruption. The network interruption information of the target device is obtained, which includes audio and video data transmitted to the target device by other target devices during the network interruption time. Based on the network interruption information of the target device, the audio and video data of all other target devices after the network interruption are temporarily stored. If the network connection of the target device is detected to be restored, the network interruption information is sent to the target device. After the target device restores its network connection, the current network parameters and environmental parameters of the target device are obtained. The comprehensive tag value of the current target device is calculated based on the parameter values ​​corresponding to each parameter. The audio and video transmission mode of the target device is determined based on the comprehensive tag value. The temporarily stored audio and video data of other target devices is sent to the target device according to the audio and video transmission mode. This embodiment forms a closed loop in the case of network interruption, by temporarily storing the audio and video data of other target devices and rematching the optimal audio and video transmission mode for the target device after the network is restored. This balances data integrity, transmission stability, and user experience in the case of network interruption.

[0057] For example, if network interruption information is obtained for a target device, the audio and video data sent by other target devices are temporarily stored. If the target device restores its network connection, i.e., after the target device reconnects to the network, the current network parameters and environmental parameters of the target device are obtained, and the parameter values ​​corresponding to each parameter are normalized. The normalized results are: network stability 0.5, bandwidth parameter 0.6, encoding / decoding capability 0.7, load capacity 0.6, and environmental interference information 0.3. Thus, the performance parameter value of the target device is determined to be 0.66, the network parameter value to be 0.54, and the environmental parameter value to be 0.3. Based on the parameter values ​​corresponding to each parameter and the preset weights of each parameter value, the current comprehensive tag value of the target device is calculated as: 0.66×30%+0.54×60%+0.3×10%=0.552. Based on the current comprehensive tag value, the audio and video transmission mode of the target device is selected as the second transmission mode, and the temporarily stored audio data and the temporarily stored video data at a lower resolution (lower than the preset resolution) are transmitted.

[0058] In one embodiment of this application, each user conducts a video call through their respective target devices. That is, the audio and video data transmitted between the target devices includes: dialogue data between the users corresponding to each target device, and also includes: posture data of each user. For each target device, and for target devices transmitting data using the second transmission mode, the video data transmitted between the target devices includes: image data of each user. The audio and video data transmission method further includes:

[0059] The system receives audio data and first image data sent by the target device when the posture change prediction information meets the first condition. The first image data is the image data corresponding to the user. The posture change prediction information represents the user's posture change trend in the future first preset time period. The first condition is that the user's posture change trend in the future first preset time period is within a preset change range.

[0060] Transmit the audio data and first image data sent by the target device to other target devices.

[0061] In this embodiment, if the posture change prediction information of the target device meets the first condition, and the target device automatically determines its load as the first condition, then audio data and first image data are sent. Based on the posture change prediction information of the target device, and based on the device sensors of each target device, combined with computer vision technology, real-time capture of dynamic features such as user limb movements and facial expressions is achieved. The posture change prediction information corresponding to the user on the target device is predicted based on the user's posture over a current period. If the user's posture change trend within a preset time period is within a preset range, then audio data and first image data are sent through the target device, where the first image data is the image data corresponding to the user. Specifically, the posture change prediction information of the target device is obtained by: capturing the user's limbs and face in real-time using the target device's built-in camera, or obtaining the user's posture change information using sensors, and extracting the user's key points using a lightweight visual algorithm. Key points can be the coordinates of the shoulder, elbow, and wrist joints, the degree of torso tilt, and the head rotation angle, etc. In this embodiment, a preset posture stillness threshold is set as follows: the rotation angle of the key points is less than 30°, the displacement amplitude per second is less than or equal to 15 pixels, and the torso tilt angle is less than or equal to 5°. Based on the user's posture change trajectory over the past 10 seconds, historical time series data is generated to determine short-term change patterns. If the posture change trajectory shows a head rotation angle increasing by 5° per second, it is predicted that the angle may increase to 15° in the next second. If the torso tilt angle remains stable at 85°±3° within 10 seconds, it is predicted that the future posture will maintain slight fluctuations. The historical time series data is input into a time series prediction model (e.g., a long short-term memory neural network), which outputs the posture change trend for the next 1-3 seconds. The rotation angle, displacement amplitude, and movement frequency of key points in this posture change trend are compared with a preset posture stillness threshold to determine whether the user's posture change trend in the first preset time period is within a preset range.

[0062] The first preset time period is set to 3 seconds. Within this first preset time period, the coordinate changes of the user's key points are compared to determine the user's displacement amplitude per unit time. The number of movements of the key points per unit time, such as the number of hand gestures or the angle of head rotation, is counted to determine the user's action frequency per unit time. Based on the user's displacement amplitude and action frequency per unit time, the user's posture change prediction information within the preset first time period is determined. If the user's posture change prediction information meets a first condition, audio data and first image data are sent through the target device. The first image data is the image data corresponding to the user. The first condition is that the user's posture change trend within the future first preset time period is within a preset range. If the user's posture change prediction information does not meet the first condition, data transmission is performed according to the second transmission mode.

[0063] This embodiment further reduces the bandwidth occupied by the transmitted video images for the target device in the second transmission mode, avoiding stuttering and bandwidth waste caused by transmitting redundant video, and ensuring the smoothness and stability of dynamic images.

[0064] In one embodiment of this application, each user conducts a video call through their respective target devices. That is, the audio and video data transmitted between the target devices includes: audio data between the users corresponding to each target device, and also includes: interface image data of each target device. For each target device, and for target devices transmitting data using the second transmission mode, the video data transmitted between the target devices includes: interface image data of each target device. The audio and video data transmission method further includes:

[0065] When the interface prediction information meets the second condition, the target device sends audio data and second image data. The second image data is the interface image data corresponding to the target device. The interface prediction information represents the interface change trend of the target device in the future second preset time period. The second condition is that the interface change trend of the target device in the future second preset time period is within the preset interface change range.

[0066] Transmit the audio data and second image data sent by the target device to other target devices.

[0067] In this embodiment, the interface change trend of the target device within a future second prediction time period is predicted based on the target device's interface prediction information. If the target device is a computer device, the coordinate positions of the mouse or cursor on the display interface are collected in real time and the coordinate change trajectory is recorded. If the target device is a mobile device, such as a tablet or mobile phone, the coordinate positions of touch points on the display interface are collected in real time through the touch screen contact points to simulate cursor movement and record the touch point change trajectory. The interface prediction information of the target device is predicted based on the coordinate change trajectory or touch point change trajectory. For computer devices, mouse coordinates are obtained in real time through the computer system API and recorded every 10ms to form a continuous trajectory, which includes straight lines, curves, and reciprocating motion. For tablet devices, single-finger and multi-finger operations are distinguished, and the physical coordinates of the finger contact points are collected through the touch screen driver, the physical coordinates are converted into interface relative coordinates, and the sliding trajectory is recorded.

[0068] Basic and key features are extracted from continuous or sliding trajectories. Basic features include direction of motion, average speed, and distribution of rest points, while key features include trajectory shape and operation rhythm. Based on these extracted features, the association patterns between each feature and the position of interface elements are determined. The type information, corresponding coordinate range, and functional description of different types of interface elements (e.g., buttons, input boxes) are associated and bound to construct an element library of element type-coordinate range-functional description. According to the matching rules between the trajectory and elements in this element library, the interface elements pointed to or involved by the trajectory are determined. Element types include buttons, input boxes, and menus. These matching rules are used to determine whether there is a correlation between trajectory features and interface elements in the element library.

[0069] The system acquires the current interface layout of the target device, identifies the types of interface elements, and their coordinate ranges (e.g., the coordinate range of a save button is x:500-550, y:20-40). It records the position, function description (e.g., print button, drop-down menu), and historical interaction frequency of the current interface elements. Based on the position of the current interface elements and the historical interaction frequency, it determines the predicted trajectory. The predicted trajectory is then matched against an element library to determine the positional relationship between the predicted trajectory and the coordinate ranges of each interface element in the library. Based on this positional relationship and the historical interaction frequency, it determines the interface element most likely associated with the predicted trajectory.

[0070] The interface change trend is determined based on the functional description of the interface element most likely associated with the predicted trajectory. If the interface change trend is within the preset interface change range, it is determined that the interface prediction information of the target device meets the second condition.

[0071] If the interface prediction information of the target device within a second preset time period meets the second condition, then the audio data and second image data sent by the target device are received, and the audio data and second image data sent by the target device are transmitted to other target devices. The second condition is that the interface prediction information of the target device within the second preset time period is within a preset range of change, and the second image data is the interface information corresponding to the target device. If the interface prediction information of the target device does not meet the second condition, then data transmission is performed according to the second transmission mode.

[0072] This embodiment targets the target device in the second transmission mode and transmits audio data and second image data to other target devices, further reducing the bandwidth occupied by the transmitted video images, avoiding stuttering and bandwidth waste caused by transmitting redundant video, and ensuring the smoothness and stability of dynamic images.

[0073] Corresponding to the audio and video data transmission method in the above embodiments, Figure 2 This is a structural block diagram of an audio / video data transmission system provided according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The audio and video data transmission system 20 includes: a device information acquisition module 21, a comprehensive tag value acquisition module 22, and a transmission mode determination module 23.

[0074] Among them, the device information acquisition module 21 is used to acquire the parameter information corresponding to each target device. Each target device is a device to be transmitted audio and video. The parameter information of a target device includes multiple parameters and their corresponding parameter values. The multiple parameters include performance parameters, network parameters and environmental parameters.

[0075] The comprehensive tag value acquisition module 22 is used to calculate the comprehensive tag value of each target device based on the parameter values ​​and weights corresponding to each parameter value. The comprehensive tag value represents the overall adaptability of the target device in audio and video data transmission, and the comprehensive tag value is positively correlated with the adaptability.

[0076] The transmission mode determination module 23 is used to determine the corresponding audio and video transmission mode of each target device according to the comprehensive tag value of each target device, so that each target device can transmit data based on its corresponding audio and video transmission mode. The audio and video transmission mode includes a first transmission mode, a second transmission mode, or a third transmission mode. The first and second transmission modes can transmit audio data and video data to the device. The resolution of the video data transmitted in the first transmission mode is higher than the preset resolution. The resolution of the video data transmitted in the second transmission mode is not higher than the preset resolution. The third transmission mode can only transmit audio data to the device.

[0077] In one embodiment of this application, the comprehensive tag value acquisition module 22 is specifically used to determine the fault type of the target device based on each parameter value. The fault type includes network fluctuation, network interruption, or sudden drop in device performance.

[0078] Set the weights for each parameter according to the fault type;

[0079] The comprehensive label value of the target device is calculated based on each parameter value and its corresponding weight.

[0080] In one embodiment of this application, the integrated tag value acquisition module 22 is further used to determine the fluctuation level corresponding to the current network fluctuation based on the parameter value corresponding to the network parameter, according to the fault type of the target device being network fluctuation. The fluctuation level is a first-level fluctuation level, a second-level fluctuation level, or a third-level fluctuation level, with the fluctuation intensity corresponding to the first-level fluctuation level, the second-level fluctuation level, and the third-level fluctuation level increasing sequentially.

[0081] The weights of each parameter are set according to the fault type. If the network fluctuation level is Level 1, the weights of each parameter value are set according to the first allocation mode.

[0082] If the network fluctuation level is Level 1, then the weights of each parameter value are set according to the second allocation mode.

[0083] If the network fluctuation level is level three, then the weights of each parameter value are set according to the third allocation mode.

[0084] Among them, the network parameter weights corresponding to the first allocation mode, the second allocation mode, and the third allocation mode decrease sequentially.

[0085] In one embodiment of this application, the integrated tag value acquisition module 22 is further used to determine the network interruption time of the target device based on the fault type of the target device being network interruption;

[0086] If the network interruption time of the target device exceeds the preset interruption time, the network interruption information of the target device is obtained. The network interruption information includes: audio and video data transmitted to the target device by other target devices during the network interruption time.

[0087] If the network connection to the target device is detected to have been restored, a network interruption message will be sent to the target device.

[0088] In one embodiment of this application, for each target device, and for a target device transmitting data in the second transmission mode, the transmission mode determination module 23 is further configured to:

[0089] The system receives audio data and first image data sent by the target device when the posture change prediction information meets the first condition. The first image data is the image information corresponding to the user. The posture change prediction information represents the user's posture change trend in the future first preset time period. The first condition is that the user's posture change trend in the future first preset time period is within a preset change range.

[0090] Transmit the audio data and first image data sent by the target device to other target devices.

[0091] In one embodiment of this application, for each target device, and for a target device transmitting data in the second transmission mode, the transmission mode determination module 23 is further configured to:

[0092] When the interface prediction information meets the second condition, the target device sends audio data and second image data. The second image data is the interface information corresponding to the target device. The interface prediction information represents the interface change trend of the target device in the future second preset time period. The second condition is that the interface change trend of the target device in the future second preset time period is within the preset interface change range.

[0093] Transmit the audio data and second image data sent by the target device to other target devices.

[0094] In one embodiment of this application, the comprehensive tag value acquisition module 22 is further used to normalize each parameter value corresponding to the target device to obtain their respective normalization results;

[0095] The comprehensive tag value of the target device is calculated based on each parameter value and its corresponding weight, including:

[0096] The comprehensive label value of the target device is determined by each normalization result and its corresponding weight.

[0097] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3The electronic device 300 in this embodiment can be the server shown above, and may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memory 304 stores computer programs, including program instructions. The processor 301 executes the program instructions stored in the memory 304. The processor 301 is configured to invoke the program instructions to perform the functions of each module / unit in the above system embodiments, for example... Figure 2 The functions of the device information acquisition module 21, the comprehensive tag value acquisition module 22, and the transmission mode determination module 23 are shown.

[0098] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0099] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0100] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store parameter information of each target device and corresponding integrated tag values.

[0101] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in the audio and video data transmission methods provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0102] In another embodiment of this application, a computer storage medium is provided. The computer storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or system capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0103] The computer storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the computer storage medium can include both internal and external storage units of the electronic device. The computer storage medium is used to store computer programs and other programs and data required by the electronic device. The computer storage medium can also be used to temporarily store data that has been output or will be output.

[0104] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0106] In the embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules, or it may be an electrical, mechanical, or other form of connection.

[0107] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0108] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio / video data transmission method, characterized by, The method comprises: obtaining parameter information corresponding to each target device, wherein the target device is a device to be subjected to audio and video transmission, the parameter information of one target device comprises a plurality of parameters and respective corresponding parameter values, and the plurality of parameters comprise performance parameters, network parameters and environmental parameters; for each target device, determining a fault type of the target device according to the parameter values corresponding to the target device, wherein the fault type comprises network fluctuation, network interruption or device performance drop; if the fault type of the target device is network fluctuation, determining a network fluctuation level corresponding to the current network fluctuation based on the parameter values corresponding to the network parameters; if the network fluctuation level is a first fluctuation level, setting the respective corresponding weights of the parameter values according to a first distribution mode; if the network fluctuation level is a second fluctuation level, setting the respective corresponding weights of the parameter values according to a second distribution mode; if the network fluctuation level is a third fluctuation level, setting the respective corresponding weights of the parameter values according to a third distribution mode; wherein the fluctuation intensity corresponding to the first fluctuation level, the second fluctuation level and the third fluctuation level increases in turn; and the network parameter weights corresponding to the first distribution mode, the second distribution mode and the third distribution mode decrease in turn; calculating a comprehensive label value of the target device according to the parameter values corresponding to the target device and the respective corresponding weights of the parameter values, wherein the comprehensive label value represents the overall adaptation capability of the target device in audio and video data transmission, and the comprehensive label value is positively correlated with the adaptation capability; determining an audio and video transmission mode corresponding to each target device according to the respective corresponding comprehensive label values of the target devices, so that each target device performs data transmission based on the respective corresponding audio and video transmission mode, wherein the audio and video transmission mode comprises a first transmission mode, a second transmission mode or a third transmission mode; obtaining historical time series data of a user corresponding to a target device, inputting the historical time series data into a time series prediction model, and determining a posture change of the user in a future first preset time period; for each target device, for the target device performing data transmission in the second transmission mode, the video data transmitted between the target devices comprises image data of respective users, and the method further comprises: receiving audio data and first picture data sent by the target device when posture change prediction information meets a first condition, wherein the first picture data is image data corresponding to a user of the target device, the posture change prediction information represents a posture change trend of the user in a future first preset time period, and the first condition is that the posture change trend of the user in the future first preset time period is within a preset change range; transmitting the audio data and the first picture data sent by the target device to other target devices; The first transmission mode and the second transmission mode can be used to transmit audio data and video data for the device, the video data transmitted by the first transmission mode has a resolution higher than a preset resolution, the video data transmitted by the second transmission mode has a resolution not higher than the preset resolution, and the third transmission mode can be used to transmit only audio data for the device.

2. The audio / video data transmission method of claim 1, wherein, If the fault type of the target device is network interruption, the method further comprises: determining a network interruption time of the target device; if the network interruption time of the target device exceeds a preset interruption time, obtaining network interruption information of the target device, the network interruption information comprising audio and video data transmitted by other target devices to the target device within the network interruption time of the target device; if it is detected that the target device is connected to the network, sending the network interruption information to the target device.

3. The audio / video data transmission method of claim 1, wherein, For each target device, for the target device transmitting data in the second transmission mode, the video data transmitted between the target devices comprises interface image data of the target devices, and the method further comprises: receiving audio data and second picture data sent by the target device when interface prediction information meets a second condition, the second picture data being interface image data corresponding to the target device, the interface prediction information representing an interface change trend of the target device in a second preset time period in the future, and the second condition being that the interface change trend of the target device in the second preset time period in the future is within a preset interface change range; transmitting the audio data and the second picture data sent by the target device to other target devices.

4. The audio / video data transmission method of claim 1, wherein, The method further comprises: normalizing each parameter value corresponding to the target device to obtain a respective normalized result; wherein the calculation of the comprehensive label value of the target device according to the parameter values and the weights corresponding to the parameter values comprises: determining the comprehensive label value of the target device through the respective normalized results and the respective corresponding weights.

5. An audio / video data transmission system, characterized by comprising: comprises: a device information acquisition module configured to acquire parameter information corresponding to each target device, the target devices being devices to be used for audio and video transmission, the parameter information of a target device comprising a plurality of parameters and respective parameter values, the plurality of parameters comprising performance parameters, network parameters and environmental parameters; a comprehensive label value acquisition module configured to, for each target device, determine a fault type of the target device according to parameter values corresponding to the target device, the fault type comprising network fluctuation, network interruption or device performance drop; if the fault type of the target device is network fluctuation, determining a network fluctuation level corresponding to the current network fluctuation based on the parameter values corresponding to the network parameters; if the network fluctuation level is a first fluctuation level, setting the respective weights corresponding to the parameter values according to a first allocation mode; if the network fluctuation level is a second fluctuation level, setting the respective weights corresponding to the parameter values according to a second allocation mode; if the network fluctuation level is a third fluctuation level, setting the respective weights corresponding to the parameter values according to a third allocation mode; The first fluctuation level, the second fluctuation level, and the third fluctuation level correspond to fluctuation intensities that increase in turn. The target device is determined based on the comprehensive label value of each target device. The transmission mode determination module is configured to determine an audio-video transmission mode corresponding to each target device based on the comprehensive label value of each target device, so that each target device transmits data based on the audio-video transmission mode corresponding thereto. The historical time series data of the user corresponding to the target device is obtained, and the historical time series data is input into a time series prediction model to determine the posture change of the user in a future first preset time period. For each target device, for the target device that transmits data in the second transmission mode, the video data transmitted between the target devices includes image data of the respective users. The transmission mode determination module is further configured to: receive audio data and first picture data transmitted by the target device when the posture change prediction information meets a first condition, the first picture data being image data corresponding to the user of the target device; the posture change prediction information represents a posture change trend of the user in a future first preset time period, and the first condition is that the posture change trend of the user in the future first preset time period is within a preset change range. transmit the audio data and the first picture data transmitted by the target device to other target devices.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The first transmission mode and the second transmission mode can transmit audio data and video data to the device, the resolution of the video data transmitted by the first transmission mode is higher than a preset resolution, the resolution of the video data transmitted by the second transmission mode is not higher than the preset resolution, and the third transmission mode can only transmit audio data to the device.

7. A computer storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 4.

Images