Portable live broadcast terminal control method and device fusing multi-source audio and video synchronous processing

Abstract

The invention is suitable for the technical field of intelligent video scheduling and remote teaching resource management, and provides a portable live broadcast terminal control method and device fusing multi-source audio and video synchronous processing. The method comprises the following steps: detecting a network state change and predicting a condition that a weak network state will be entered in a future preset time period and the duration exceeds a preset threshold value; according to the invention, by introducing the simulation weakening experiment and the comprehensive performance deviation analysis mechanism, the adaptive priority adjustment of the multi-source video acquisition content in the prediction weak network state is realized, and the limitations of static priority configuration and lack of learning behavior perception in the prior art are broken through. The system can identify acquisition sources which are more sensitive to learning influence in teacher explanation and practical training equipment pictures, generate correction factors based on user actual responses, dynamically optimize resolution regulation priority values, and improve the reasonability of video scheduling strategies and the fidelity of teaching pictures in a weak network environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent video scheduling and remote teaching resource management, and in particular relates to a portable live broadcast terminal control method and device integrating multi-source audio and video synchronous processing. Background Art

[0002] Existing live broadcast systems for remote teaching or practical training usually operate based on a multi-source video acquisition architecture, which includes the synchronous transmission of multiple types of visual content such as teacher explanations and demonstration equipment operation processes. To ensure the transmission effect of video streams in multi-terminal and multi-network environments, some systems have introduced video source priority control strategies based on preset weights or fixed rules to enable resource compression scheduling when bandwidth is limited or network fluctuations occur. However, this type of scheduling logic mostly relies on static configuration parameters and lacks the ability to dynamically identify and regulate the learning status of the training group in the current teaching situation. As a result, images that are more critical to teaching effectiveness may be incorrectly compressed under weak network conditions, affecting learning continuity and knowledge transfer efficiency.

[0003] Existing technologies for predicting and responding to weak network conditions primarily rely on bandwidth monitoring models, packet loss rate statistics, or network sliding window prediction algorithms to make judgments and, accordingly, trigger policy control behaviors such as downsizing and frame reduction. However, the core execution units of most of these policies fail to incorporate the semantic structure of the teaching content or actual user feedback, relying solely on content type tags or system configuration files to set compression weights. This makes it difficult to accurately identify the true visual focus and cognitive dependency elements in different teaching tasks. Furthermore, existing systems generally lack an effective evaluation mechanism based on learning-based sensitivity to determine which source of video should be prioritized for clarity. Summary of the Invention

[0004] The purpose of the present invention is to provide a portable live broadcast terminal control method and device that integrates multi-source audio and video synchronous processing, aiming to solve the problems raised in the background technology.

[0005] The present invention is implemented as follows: a portable live broadcast terminal control method integrating multi-source audio and video synchronous processing, the method comprising:

[0006] When a network status change is detected and it is predicted that a weak network state will be entered within a predetermined period of time in the future and the duration exceeds a preset threshold, a simulated weakening experiment is performed on the teacher's explanation video acquisition source and the demonstration device video acquisition source. The simulated weakening experiment includes:

[0007] The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged.

[0008] In the second weakening experiment, the resolution of the video source of the demonstration equipment was reduced, while the resolution of the video source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent;

[0009] Collecting the comprehensive performance of the training group during the two weakening experiments, determining the weakening experiment with the poorer comprehensive performance, and determining the video acquisition source with the reduced resolution in the weakening experiment as the target video acquisition source;

[0010] Obtaining the actual comprehensive performance of the training group within a preset period before prediction, and generating a correction factor based on the deviation between the actual comprehensive performance and the comprehensive performance of the weakening experiment corresponding to the target video acquisition source;

[0011] Get the initial resolution control priority value generated for the target video acquisition source and correct it using the correction factor.

[0012] As a further limitation of the technical solution of the embodiment of the present invention, the duration of the first weakening experiment and the second weakening experiment are both the same as the preset time period, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period.

[0013] As a further limitation of the technical solution of the embodiment of the present invention, after detecting a change in network status and predicting that a weak network state will be entered within a predetermined period in the future and the duration will exceed a preset threshold, the predicted resolution reduction corresponding to the teacher's explanation video acquisition source and the demonstration equipment video acquisition source under the weak network state is analyzed; in the first weakening experiment and the second weakening experiment, the resolution reduction adopted for the teacher's explanation video acquisition source or the demonstration equipment video acquisition source is consistent with the predicted resolution reduction of the corresponding video acquisition source under the weak network state.

[0014] As a further limitation of the technical solution of the embodiment of the present invention, the comprehensive performance effect is the overall performance status of the trainee group in the process of receiving the teaching content, and the comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

[0015] As a further limitation of the technical solution of the embodiment of the present invention, the steps of obtaining the initial resolution control priority value generated for the target video acquisition source and correcting it using the correction factor include:

[0016] Obtaining an initial resolution control priority value set for a target video acquisition source and a preset correction formula for adjusting the initial resolution control priority value;

[0017] Substituting the correction factor into a preset correction formula to correct the initial resolution control priority value, thereby obtaining a corrected resolution control priority value;

[0018] The revised resolution control priority value is applied to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to realize the dynamic adjustment of the resource allocation strategy.

[0019] As a further limitation of the technical solution of the embodiment of the present invention, the priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

[0020] A portable live broadcast terminal control device integrating multi-source audio and video synchronous processing, the device comprising: an experiment execution module, a target video acquisition source determination module, a correction factor generation module, and a priority value correction module, wherein:

[0021] The experiment execution module is used to perform a simulated weakening experiment on the teacher's explanation video acquisition source and the demonstration device video acquisition source when a network status change is detected and it is predicted that the network will enter a weak network state within a predetermined period of time in the future and the duration exceeds a preset threshold. The simulated weakening experiment includes:

[0022] The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged.

[0023] In the second weakening experiment, the resolution of the video source of the demonstration equipment was reduced, while the resolution of the video source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent;

[0024] The target video acquisition source determination module is used to collect the comprehensive performance of the training group during the two weakening experiments, determine the weakening experiment with the poorer comprehensive performance, and determine the video acquisition source with the lowered resolution in the weakening experiment as the target video acquisition source;

[0025] a correction factor generation module, configured to obtain an actual comprehensive performance effect of the training group within a preset time period before prediction, and generate a correction factor based on a deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source; the duration of the first weakening experiment and the second weakening experiment are both the same as the preset time period, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period;

[0026] The priority value correction module is used to obtain the initial resolution control priority value generated for the target video acquisition source and correct it using a correction factor.

[0027] As a further limitation of the technical solution of the embodiment of the present invention, after detecting a change in network status and predicting that a weak network state will be entered within a predetermined period in the future and the duration will exceed a preset threshold, the predicted resolution reduction corresponding to the teacher's explanation video acquisition source and the demonstration equipment video acquisition source under the weak network state is analyzed; in the first weakening experiment and the second weakening experiment, the resolution reduction adopted for the teacher's explanation video acquisition source or the demonstration equipment video acquisition source is consistent with the predicted resolution reduction of the corresponding video acquisition source under the weak network state.

[0028] As a further limitation of the technical solution of the embodiment of the present invention, the comprehensive performance effect is the overall performance status of the trainee group in the process of receiving the teaching content, and the comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

[0029] As a further limitation of the technical solution of the embodiment of the present invention, the priority value correction module specifically includes:

[0030] A data acquisition unit, configured to acquire an initial resolution control priority value set for a target video acquisition source, and a preset correction formula for adjusting the initial resolution control priority value;

[0031] a priority value correction unit, configured to substitute a correction factor into a preset correction formula to correct the initial resolution control priority value, thereby obtaining a corrected resolution control priority value;

[0032] A modified priority value application unit is used to apply the modified resolution control priority value to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to realize dynamic adjustment of the resource allocation strategy;

[0033] The priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] By introducing simulated weakening experiments and comprehensive performance deviation analysis mechanisms, the present invention achieves adaptive priority adjustment of multi-source video acquisition content under predicted weak network conditions, breaking through the limitations of static priority configuration and lack of learning behavior perception in existing technologies. The system can identify acquisition sources that are more sensitive to learning impacts in teacher explanations and training equipment images, and generate correction factors based on actual user responses, dynamically optimize resolution and control priority values, and improve the rationality of video scheduling strategies and the fidelity of teaching images in weak network environments. This solution has the advantages of strong pertinence, clear computational structure, and low deployment threshold, and is suitable for refined image resource scheduling in diverse remote teaching scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart of modifying the initial resolution control priority value in the method provided in an embodiment of the present invention;

[0038] Figure 3 An application architecture diagram of the system provided by an embodiment of the present invention;

[0039] Figure 4 This is a structural block diagram of a priority value correction module in a system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] Figure 1 A flow chart of a method provided by an embodiment of the present invention is shown.

[0042] Specifically, a portable live broadcast terminal control method integrating multi-source audio and video synchronous processing includes the following steps:

[0043] Step S100: When a network status change is detected and it is predicted that a weak network state will be entered within a predetermined period of time in the future and the duration exceeds a preset threshold, a simulated weakening experiment is performed on the teacher's lecture video acquisition source and the demonstration device video acquisition source. The simulated weakening experiment includes:

[0044] The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged.

[0045] In the second weakening experiment, the resolution of the video acquisition source of the demonstration equipment was reduced, while the resolution of the video acquisition source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent.

[0046] The durations of the first weakening experiment and the second weakening experiment are the same, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period.

[0047] After detecting a change in network status and predicting that a weak network state will be entered within a predetermined period in the future and the duration will exceed a preset threshold, the predicted resolution reduction corresponding to the teacher's explanation video acquisition source and the demonstration equipment video acquisition source under the weak network state is analyzed; in the first weakening experiment and the second weakening experiment, the resolution reduction adopted for the teacher's explanation video acquisition source or the demonstration equipment video acquisition source is consistent with the predicted resolution reduction of the corresponding video acquisition source under the weak network state.

[0048] In the teaching scenario implemented in this embodiment, the system is configured to provide two parallel information sources: teacher explanations and hands-on equipment demonstrations. These are independently captured and synchronously processed by two video acquisition sources. Throughout the teaching process, the trainee relies on both types of visual information for cognitive and operational judgment, neither of which can be disregarded. Furthermore, the device types corresponding to the two video acquisition sources are essentially identical in structure and performance, providing comparability and equivalent adjustment capabilities, which forms the basis for subsequent experimental control and priority adjustment logic.

[0049] For weak network environments, the present invention presupposes that: when the system detects a change in the network state, and predicts through bandwidth prediction models, network fluctuation trend algorithms, or delay gradient calculations that the network will enter a weak network state within a certain period of time in the future, and the duration of this state will exceed the preset threshold set by the platform, the teaching system is triggered to enter the parameter adaptive control preparation stage. The above-mentioned detection and prediction technical means currently belong to the scope of existing technologies. For example, combining historical transmission packet loss rate sequences, bandwidth prediction models based on sliding windows, or timing judgment algorithms based on a combination of moving average and exponential weighting are all well-known methods in the art. Therefore, in the present invention, the detection of a weak network itself is not considered an innovative point, but rather a prerequisite for triggering the judgment strategy experiment.

[0050] In existing technologies, when a weak network state is detected, the system typically generates corresponding resolution control priority values ​​for the teacher's lecture video source and the demonstration device video source based on historical bandwidth usage, task preset weight information, content tag classification, or default policy configuration. This resolution control priority value is used to guide video scheduling strategies under conditions of limited bandwidth and insufficient system resources, determining the clarity guarantee level of each video source in the overall output stream. Specifically, it determines which frame's high-resolution encoding is prioritized during dynamic compression, or which acquisition source's encoding quality parameters are prioritized for multi-channel video fusion transcoding.

[0051] Such priority values ​​are often statically generated or rule-driven, without considering the actual content dependency structure of the current training group or the actual feedback received in the current teaching scenario. This can easily lead to weight bias, resulting in insufficient priority retention of key learning scenes, which affects the overall teaching effect. Therefore, it is necessary to modify the initial resolution control priority value through the simulated weakening experiment and performance deviation evaluation mechanism proposed in this invention, so that it has the ability to perceive real scenes and adapt to the learning experience, thereby improving the rationality of the system scheduling strategy and the dynamic response effect.

[0052] This preliminary analysis of the predicted resolution reduction for two video sources under future weak network conditions ensures engineering equivalence for subsequent simulated degradation experiments. This means that the compression conditions applied during the experiment must be consistent with the compression conditions the system anticipates will actually be applied, preventing experimental results from deviating from future reality and ensuring the collected responses from the trained group are valuable references. This analysis allows the system to predict the compression ratio or resolution reduction level to be applied under weak network conditions based on the encoding complexity, current code control configuration, and device performance limits of each video source, combined with the current network model, thereby establishing a corresponding set of predicted reduction parameters.

[0053] The implementation methods of the first weakening experiment and the second weakening experiment set up by the present invention are to perform resolution reduction processing on a single video acquisition source, and keep the other video acquisition source at the current image quality state, thereby forming a single-factor control structure of "only weakening one variable" in each experiment, so that the experimental results have clear attributability. In this way, it is possible to effectively determine which of the two video acquisition sources has a more significant impact on the trainees' learning performance due to the reduction in picture clarity, thereby providing a reliable judgment basis for the correction of priority parameters. Furthermore, the pressure reduction amplitude is required to be consistent with the compression amplitude that the system expects to take for the acquisition source under future network prediction conditions, in order to maintain the authenticity of the experiment and the matching of the previous and subsequent application strategies, to avoid the imbalance phenomenon of "light compression in experiment / heavy compression in reality" or "heavy compression in experiment / light compression in reality", and to ensure that the correction logic is effective in a real environment.

[0054] In addition, the duration of the two weakening experiments is set to be the same, and the total experimental time is less than the limit of the predicted weak network period duration. This is mainly based on the following two considerations: first, to ensure the balance of the time coverage of the two experimental segments, so as to eliminate the interference of psychological load, fatigue or attention changes caused by the difference in time length on the trainees' performance, and ensure the comparability of the results; second, to control the total experimental time within the predicted weak network time, to ensure that the experimental process is completed before the implementation of the system control strategy, to avoid affecting the execution window of the system's actual response strategy, and to achieve an orderly connection between experimental sampling and real-time scheduling. This setting helps the system complete the priority adjustment decision before the weak network state actually arrives, reflecting the pre-control capability and response efficiency of the method of the present invention.

[0055] Furthermore, the portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing further includes the following steps:

[0056] Step S200 , collecting the comprehensive performance effects of the training group during the two weakening experiments, determining the weakening experiment with the poorer comprehensive performance effect, and determining the video acquisition source with the reduced resolution in the weakening experiment as the target video acquisition source.

[0057] Step S300 : Obtaining the actual comprehensive performance of the training group within a preset period before prediction, and generating a correction factor based on the deviation between the actual comprehensive performance and the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. The duration of the first weakening experiment and the second weakening experiment are both the same as the preset period.

[0058] The comprehensive performance effect is the overall performance status of the training group in the process of receiving the teaching content. The comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

[0059] In an embodiment of the present invention, by performing two weakening experiments, the system can respectively observe the effects of the teacher's explanation video acquisition source and the demonstration equipment video acquisition source on the actual learning performance of the trainee group when the image quality is weakened. By comparing and analyzing the comprehensive performance of the trainee group in the two weakening experiments, it can be identified which type of video acquisition source has a lower image quality that is more likely to lead to a decline in learning experience or a weakening of learning effect. Therefore, the video acquisition source with a lowered resolution in the weakening experiment is determined as the target video acquisition source in order to accurately lock in the content source that is "most sensitive to the weakening of image quality." This recognition result serves as the basis for subsequent priority corrections, which helps the system to give priority to visual content sources that have a greater impact on learning during weak network regulation, thereby improving the stability and adaptability of the overall teaching quality.

[0060] The comprehensive performance effect is used to reflect the comprehensive acceptance ability and learning status performance of the training group in the process of receiving teaching content. In the present invention, the comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include but are not limited to facial expression changes, eye movement trajectories, operation behavior frequency, note content changes and voice feedback quality. Each behavioral feature can be collected and quantified based on existing feasible technologies: facial expression changes can be realized by expression recognition algorithms based on OpenCV or Dlib; eye movement trajectories can be extracted with the help of infrared eye trackers or camera tracking combined with Gaze Tracking algorithms; operation frequency and note-taking behavior can be automatically collected by the touch behavior recording module of the terminal device; voice feedback can be based on voice recognition and emotion analysis algorithms (such as MFCC feature extraction + emotion classification network) to judge response timeliness and feedback stability. The scoring criteria for each feature are normalized in the training sample to generate weights, and finally form a comprehensive performance scoring index to ensure its feasibility, repeatability and objectivity.

[0061] In step S300, the comprehensive performance of the target video acquisition source during the suppression experiment is obtained and compared with the actual comprehensive performance of the training group under normal network conditions, and the deviation is calculated to generate a correction factor. This correction factor quantifies the "relative negative impact of the weakened image quality of the video acquisition source on the learning effect", which essentially reflects the importance of the acquisition source to the learning behavior. The introduction of the correction factor means that the initial resolution control priority value is no longer based solely on static policy settings such as task labels and historical presets, but can dynamically adapt to the actual acceptance characteristics of the current user group, thereby making parameter control more individually targeted and real-time adaptable.

[0062] Furthermore, setting the duration of the first and second weakening experiments to coincide with the preset time periods ensures consistency in the complexity and time intensity of the learning content covered in the two experimental phases. This prevents behavioral deviations such as attention lapse, psychological fatigue, or changes in operation density among the training groups due to different phase lengths, helping to ensure the comparability of experimental results. Furthermore, setting the duration to coincide with the time window corresponding to the actual comprehensive performance results obtained before prediction facilitates the system's unified sampling, normalized comparison, and correction factor construction of deviation amplitudes, enhancing the accuracy and mathematical stability of the subsequent priority correction process.

[0063] Furthermore, the portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing further includes the following steps:

[0064] Step S400: obtaining an initial resolution control priority value generated for a target video acquisition source, and correcting it using a correction factor.

[0065] Specifically, Figure 2 A flow chart for modifying the initial resolution control priority value is shown.

[0066] The steps of obtaining the initial resolution control priority value generated for the target video acquisition source and correcting it using the correction factor specifically include the following steps:

[0067] Step S401, obtaining an initial resolution control priority value set for a target video acquisition source and a preset correction formula for adjusting the initial resolution control priority value;

[0068] Step S402: Substituting the correction factor into a preset correction formula to correct the initial resolution control priority value to obtain a corrected resolution control priority value;

[0069] Step S403 : applying the corrected resolution control priority value to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to achieve dynamic adjustment of the resource allocation strategy.

[0070] The priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

[0071] In this embodiment of the present invention, to adjust the resolution control priority of the target video acquisition source under weak network conditions, the system implements a correction mechanism based on the degree of overall performance deviation. The core of this correction mechanism is the introduction of a correction factor to reflect the actual impact of the target video acquisition source's image quality degradation on the training group's learning outcomes. This correction factor modifies the initial resolution control priority values ​​set by the system based on empirical rules or default policies, effectively improving the adaptability and targeted nature of resource allocation strategies.

[0072] In the teaching scenario of the present invention, the system is set up to have two parallel information sources, namely teacher explanation and practical equipment demonstration, which are independently collected and synchronously processed by two video acquisition sources. The trainee group's dependence on these two types of images during the teaching process is parallel and complementary. The existing technology often only sets static priority values ​​based on the type label of the video acquisition source, system configuration parameters or average bandwidth occupancy, which cannot reflect the criticality of a certain acquisition source to the learning behavior in a specific teaching task. Especially under conditions where bandwidth is limited and trade-offs must be made, it is very easy to cause key content to be incorrectly compressed, affecting the overall teaching effect. The introduction of the correction factor just solves the dynamic control blind spot in this type of special application scenario that relies on strongly coupled images.

[0073] The correction formula used in the present invention is a linear correction method based on weighted deviation. Specifically, by obtaining the actual comprehensive performance of the trained group under normal network conditions and comparing it with the comprehensive performance of the target video acquisition source during the weakening experiment, the relative deviation between the two is calculated. After multiplying the degree of deviation by the adjustment coefficient, it acts as a proportional gain term on the initial resolution control priority value, and finally obtains the corrected priority value. This calculation method has the advantages of intuitive structure, simple calculation, and easy real-time deployment. It is suitable for realizing online scheduling control in a resource-constrained portable live terminal environment.

[0074] Although a linear proportional correction formula is currently used, the correction mechanism is not limited to this expression, and can also be expanded to a more complex nonlinear correction model. For example, an exponential correction function can be used to make the priority change smoothly when the degree of deviation is small, and the priority is quickly increased after the deviation reaches a critical value; a piecewise function or a logical threshold model can also be introduced to preset multiple correction thresholds according to different teaching contents or situations to achieve condition-triggered jump control. In addition, a regression learning model or reinforcement learning strategy based on sample feedback data can also be constructed to further improve the adaptability and personalization level of the scheduling strategy by continuously updating the optimization correction factor generation rules. The above-mentioned variations can all be regarded as equivalent implementations within the scope of protection of the present invention.

[0075] Further, Figure 3 The application architecture diagram of the system provided by the embodiment of the present invention is shown.

[0076] Among them, in another preferred embodiment provided by the present invention, a portable live terminal control device integrating multi-source audio and video synchronous processing includes:

[0077] The experiment execution module 100 is configured to execute a simulated weakening experiment on the teacher's explanation video acquisition source and the demonstration device video acquisition source when a network status change is detected and it is predicted that the network will enter a weak network state within a predetermined period of time in the future and the duration exceeds a preset threshold. The simulated weakening experiment includes:

[0078] The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged.

[0079] In the second weakening experiment, the resolution of the video acquisition source of the demonstration equipment was reduced, while the resolution of the video acquisition source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent.

[0080] After detecting a change in network status and predicting that a weak network state will be entered within a predetermined period in the future and the duration will exceed a preset threshold, the predicted resolution reduction corresponding to the teacher's explanation video acquisition source and the demonstration equipment video acquisition source under the weak network state is analyzed; in the first weakening experiment and the second weakening experiment, the resolution reduction adopted for the teacher's explanation video acquisition source or the demonstration equipment video acquisition source is consistent with the predicted resolution reduction of the corresponding video acquisition source under the weak network state.

[0081] Furthermore, the portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing also includes:

[0082] The target video acquisition source determination module 200 is used to collect the comprehensive performance effects of the training group during the two weakening experiments, determine the weakening experiment with the worse comprehensive performance effect, and determine the video acquisition source with the lowered resolution in the weakening experiment as the target video acquisition source.

[0083] The comprehensive performance effect is the overall performance status of the training group in the process of receiving the teaching content. The comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

[0084] Furthermore, the portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing also includes:

[0085] The correction factor generation module 300 is used to obtain the actual comprehensive performance effect of the training group within a preset time period before prediction, and generate a correction factor based on the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source; the duration of the first weakening experiment and the second weakening experiment are both the same as the preset time period, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period.

[0086] Furthermore, the portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing also includes:

[0087] The priority value correction module 400 is used to obtain the initial resolution control priority value generated for the target video acquisition source and correct it using a correction factor.

[0088] Specifically, Figure 4 FIG. 4 is a structural block diagram of a priority value correction module 400 in a system provided by an embodiment of the present invention.

[0089] In a preferred embodiment of the present invention, the priority value correction module 400 specifically includes:

[0090] The data acquisition unit 401 is used to obtain an initial resolution control priority value set for a target video acquisition source and a preset correction formula for adjusting the initial resolution control priority value;

[0091] The priority value correction unit 402 is used to substitute the correction factor into a preset correction formula to correct the initial resolution control priority value to obtain a corrected resolution control priority value;

[0092] The modified priority value application unit 403 is configured to apply the modified resolution control priority value to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to achieve dynamic adjustment of the resource allocation strategy;

[0093] The priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

[0094] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0095] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0096] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A portable live broadcast terminal control method integrating multi-source audio and video synchronous processing, characterized in that: The method comprises: When a network status change is detected and it is predicted that a weak network state will be entered within a predetermined period of time in the future and the duration exceeds a preset threshold, a simulated weakening experiment is performed on the teacher's explanation video acquisition source and the demonstration device video acquisition source. The simulated weakening experiment includes: The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged. In the second weakening experiment, the resolution of the video source of the demonstration equipment was reduced, while the resolution of the video source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent; Collecting the comprehensive performance of the training group during the two weakening experiments, determining the weakening experiment with the poorer comprehensive performance, and determining the video acquisition source with the reduced resolution in the weakening experiment as the target video acquisition source; Obtaining the actual comprehensive performance of the training group within a preset period before prediction, and generating a correction factor based on the deviation between the actual comprehensive performance and the comprehensive performance of the weakening experiment corresponding to the target video acquisition source; Get the initial resolution control priority value generated for the target video acquisition source and correct it using the correction factor.

2. The portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing according to claim 1 is characterized in that: The durations of the first weakening experiment and the second weakening experiment are both the same as the preset time period, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period.

3. The portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing according to claim 2 is characterized in that: After detecting a change in the network state and predicting that a weak network state will occur within a predetermined period of time in the future and that the duration exceeds a preset threshold, the predicted resolution reduction corresponding to the teacher's lecture video acquisition source and the demonstration device video acquisition source under this weak network state is analyzed; In the first weakening experiment and the second weakening experiment, the resolution reduction range used for explaining the video acquisition source to the teacher or the video acquisition source of the demonstration equipment is consistent with the predicted resolution reduction range of the corresponding video acquisition source under the weak network state.

4. The portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing according to claim 3 is characterized in that: The comprehensive performance effect is the overall performance status of the training group in the process of receiving the teaching content. The comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

5. The portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing according to claim 3 is characterized in that: The steps of obtaining an initial resolution control priority value generated for a target video acquisition source and correcting it using a correction factor include: Obtaining an initial resolution control priority value set for a target video acquisition source and a preset correction formula for adjusting the initial resolution control priority value; Substituting the correction factor into a preset correction formula to correct the initial resolution control priority value, thereby obtaining a corrected resolution control priority value; The revised resolution control priority value is applied to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to realize the dynamic adjustment of the resource allocation strategy.

6. The portable live broadcast terminal control method for integrating multi-source audio and video synchronous processing according to claim 5 is characterized in that: The priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

7. A portable live broadcast terminal control device integrating multi-source audio and video synchronous processing, characterized in that: The device includes: an experiment execution module, a target video acquisition source determination module, a correction factor generation module, and a priority value correction module, wherein: The experiment execution module is used to perform a simulated weakening experiment on the teacher's explanation video acquisition source and the demonstration device video acquisition source when a network status change is detected and it is predicted that the network will enter a weak network state within a predetermined period of time in the future and the duration exceeds a preset threshold. The simulated weakening experiment includes: The first weakening experiment reduces the resolution of the teacher's video source while keeping the resolution of the demonstration device's video source unchanged. In the second weakening experiment, the resolution of the video source of the demonstration equipment was reduced, while the resolution of the video source of the teacher's explanation remained unchanged. The duration and reduction of the two weakening experiments were consistent; The target video acquisition source determination module is used to collect the comprehensive performance of the training group during the two weakening experiments, determine the weakening experiment with the poorer comprehensive performance, and determine the video acquisition source with the lowered resolution in the weakening experiment as the target video acquisition source; a correction factor generation module, configured to obtain an actual comprehensive performance effect of the training group within a preset time period before prediction, and generate a correction factor based on a deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source; the duration of the first weakening experiment and the second weakening experiment are both the same as the preset time period, and the total duration of the simulated weakening experiment is less than the duration of the future predetermined time period; The priority value correction module is used to obtain the initial resolution control priority value generated for the target video acquisition source and correct it using a correction factor.

8. The portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing according to claim 7 is characterized in that: After detecting a change in the network state and predicting that a weak network state will occur within a predetermined period of time in the future and that the duration exceeds a preset threshold, the predicted resolution reduction corresponding to the teacher's lecture video acquisition source and the demonstration device video acquisition source under this weak network state is analyzed; In the first weakening experiment and the second weakening experiment, the resolution reduction range used for explaining the video acquisition source to the teacher or the video acquisition source of the demonstration equipment is consistent with the predicted resolution reduction range of the corresponding video acquisition source under the weak network state.

9. The portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing according to claim 8 is characterized in that: The comprehensive performance effect is the overall performance status of the training group in the process of receiving the teaching content. The comprehensive performance effect is obtained by weighted calculation of at least two scorable behavioral characteristics, and the behavioral characteristics include any two or more of facial expression changes, eye movement trajectories, operation behavior frequency, note content changes, and voice feedback quality.

10. The portable live broadcast terminal control device for integrating multi-source audio and video synchronous processing according to claim 9 is characterized in that: The priority value correction module specifically includes: A data acquisition unit, configured to acquire an initial resolution control priority value set for a target video acquisition source, and a preset correction formula for adjusting the initial resolution control priority value; a priority value correction unit, configured to substitute a correction factor into a preset correction formula to correct the initial resolution control priority value, thereby obtaining a corrected resolution control priority value; A modified priority value application unit is used to apply the modified resolution control priority value to the resolution scheduling process of the target video acquisition source after entering the predicted weak network state, so as to realize dynamic adjustment of the resource allocation strategy; The priority value correction formula is: ,in Refers to the corrected resolution control priority value, Refers to the initial resolution control priority value, Refers to the actual comprehensive performance effect, Refers to the comprehensive performance of the weakening experiment corresponding to the target video acquisition source. Refers to the correction factor, that is, the deviation between the actual comprehensive performance effect and the comprehensive performance effect of the weakening experiment corresponding to the target video acquisition source. is the adjustment coefficient of the correction factor, and Greater than 0.

Images