Conference room audio and video synchronous visualization control method and system

Abstract

The invention discloses a conference room audio and video synchronous visualization control method and system, and relates to the technical field of conference audio and video control, and the method comprises the steps: extracting the conference type of an initiator of a conference room, recording the historical behavior data of a participant in the current conference type, and storing the historical behavior data in a database; calculating permission open values of all users through a dynamic weight strategy; monitoring the equipment state of each conference member to obtain an equipment response influence value; through a hierarchical compensation algorithm, a compensation algorithm is distributed for each participant in combination with the user permission open value and the equipment state; an optimal parameter combination is searched through Bayesian optimization; and setting a synchronous correction strategy, dynamically adjusting an audio and video delay tolerance threshold according to the permission open value, and adjusting the Bayesian optimization period of the user when the actual delay time is greater than the delay tolerance threshold. According to the invention, the problem that the traditional audio and video synchronization scheme cannot adjust the synchronization strategy in a personalized manner accurately in real time for participants is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of conference audio and video control, and in particular to a conference room audio and video synchronous visualization control method and system. Background Art

[0002] The problem of audio and video synchronization is not only reflected in simple audio delay or video lag, but also involves the combined effect of multiple factors, such as network bandwidth, device performance, device configuration of participants, complexity of meeting scenarios, etc. Traditional audio and video synchronization methods mostly rely on simple buffering mechanisms or fixed time delay adjustments, but these methods are often difficult to cope with the complex and changing scenarios in actual meetings. For example, when the network bandwidth fluctuates greatly, a single synchronization mechanism is likely to cause image delay or audio lag, which in turn affects the interactive experience of participants and reduces meeting efficiency. This problem is particularly prominent in multi-party meetings and frequent interactions.

[0003] Although existing video conferencing systems can provide basic audio and video synchronization functions, their compensation mechanisms are mostly based on static settings or simple buffering mechanisms, and cannot dynamically adapt to real-time changes in different users and devices. In addition, in traditional conference systems, the audio and video synchronization compensation of all participants is usually unified, and it is impossible to optimize according to the actual needs of different participants. For hosts and others, their speeches are more important and require higher synchronization accuracy. Existing technologies usually lack intelligent adjustments based on user identity factors, making it difficult to provide customized and accurate audio and video synchronization compensation solutions based on actual conditions, wasting computing resources.

[0004] To this end, the present invention provides a conference room audio and video synchronization visualization control method and system. Summary of the invention

[0005] The purpose of the present invention is to provide a conference room audio and video synchronization visualization control method and system to solve the existing problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a conference room audio and video synchronization visualization control method, comprising the following steps: S1. Extract the conference type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the permission opening value of all users through the dynamic weight strategy; S2. Monitor the device status of each conference member and obtain the device response impact value; S3. A compensation algorithm is assigned to each participant through a hierarchical compensation algorithm, combined with the user permission opening value and device status; and the optimal parameter combination is searched through Bayesian optimization; S4. Set a synchronization correction strategy, dynamically adjust the audio and video delay tolerance threshold according to the permission opening value, and when the actual delay time is greater than the delay tolerance threshold, return to S3 and adjust the Bayesian optimization cycle of the user.

[0007] The present invention is further improved in that the dynamic weight strategy integrates the three elements of role, behavior and violation, and suppresses the expansion of the rights of long-term inactive users through a logarithmic function. The specific steps include: obtaining a user behavior score based on the behavior type of each participating user ; Obtain the cumulative value of the user violation event attenuation model through the violation events of each participating user ; And then get the permission value of each participating user ,in Indicates the user's basic role constant. ,in, On behalf of the visitor, Indicates members, Indicates the host, represents the time normalization base, Indicates the constant weight of the user's basic role, represents the user behavior score weight, Represents the cumulative value weight of the user violation event attenuation model.

[0008] The present invention is further improved in that the specific step S3 comprises: S31, obtaining the total authority value Tpe of each conference participant by weighted summing up the device response impact value and the authority opening value; S32, setting a first permission threshold TPe1 and a second permission threshold TPe2, when When When It is considered a high-authority scenario. S33. A hierarchical compensation algorithm is used to realize the delay compensation requirements of different conference participants.

[0009] The present invention is further improved in that the layered compensation algorithm includes a basic delay compensation layer, an enhanced delay compensation layer and an advanced delay compensation layer; the basic delay compensation layer calculates the change of the playback rate ,in is the time difference between audio and video, is the frame time of the video, and the delay compensation of low authority is realized by changing the pitch without changing the speed; the enhanced delay compensation layer dynamically adjusts the buffer depth according to the total value of the authority, and the enhanced buffer depth , Indicates the size of the basic buffer; the advanced delay compensation layer is based on the LSTM network, dynamically adjusts the delay step and the signal path weight through the total authority value, and generates a synchronization signal.

[0010] The present invention is further improved in that the advanced delay compensation layer takes the Mel spectrum of the past M frames and the normalized permission value PTpe as input to dynamically adjust the prediction delay compensation of the model. .

[0011] The present invention is further improved in that the audio and video delay tolerance threshold is calculated by compensating the delay predicted in step S3. As a basis, calculate the actual audio and video delay tolerance threshold , is a coefficient used to adjust the sensitivity of the tolerance threshold; when the absolute value of the difference between the actual arrival time of the audio and video stream and the expected time When the delay is greater than the audio and video delay tolerance threshold, the synchronization correction strategy is triggered, and the synchronization correction strategy includes setting a reference correction period , and get the updated correction period .

[0012] The present invention is further improved in that the calculation process of the behavior score includes setting the speech duration interval, the speech frequency interval and the number of interactions, and the speech duration score is obtained by calculating the ratio of the historical average speech time of the participants of this type to the speech duration and the deviation time of the edge value. The speech frequency score is obtained in the same way. The speech duration score, speech frequency score and standardized number of interactions of users with a time window of 30 days are taken as the behavior type and sent to the user behavior data set. Get the user behavior score , where noa indicates the number of meetings of the participant within 30 days. Indicates that the participant The first Behavior type scores, Indicates The weight coefficient of each behavior type, represents the attenuation coefficient, Indicates The time when each behavior type occurs, Indicates the current time.

[0013] The present invention is further improved in that the cumulative value of the user violation event attenuation model includes determining the user violation behavior. When the user interrupts the speech greater than 0 and less than the set first interruption number threshold, it is determined to be a minor violation. When the user interrupts the speech greater than the set first interruption number threshold, it is determined to be a severe violation. The violation behavior of the participating users with a time window of 30 days is extracted to obtain the user violation event attenuation model. , Indicates the severity of the defined violation event, H=720min indicates the half-decay period, Indicates The time when the violation occurred.

[0014] A further improvement of the present invention is that the device response impact value is obtained by monitoring the average value of the historical device synchronization delay of the conference members, and calculating the device response impact value of the conference members. Delay per meeting: ; in, Indicates Meeting No. The time of audio transmission of each speech, Indicates Meeting No. The time of the speech video, Indicates the number of user meetings within a 30-day time window. and Indicates the weight of the setting.

[0015] In another aspect, the present invention provides a conference room audio and video synchronization visualization control system, comprising: The user rights opening calculation module is used to extract the conference type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the rights opening values ​​of all users through the dynamic weight strategy; The device response monitoring module is used to monitor the device status of each conference member and obtain the device response impact value; The hierarchical compensation algorithm module is used to assign a compensation algorithm to each participant based on the user permission opening value and device status; and search for the optimal parameter combination through Bayesian optimization; The synchronization correction strategy module is used to dynamically adjust the audio and video delay tolerance threshold according to the permission opening value. When the actual delay time is greater than the delay tolerance threshold, it returns to S3 and adjusts the Bayesian optimization cycle of the user.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention first divides users into different authority levels and adopts compensation algorithms that adapt to different needs. Low-authority users use simpler compensation algorithms, while high-authority users use complex compensation algorithms, so as to ensure that the synchronization effects of different users are optimally satisfied. The compensation level is dynamically adjusted according to factors such as the user's device status and authority value to ensure real-time audio and video synchronization, especially in a multi-user environment, to ensure that each user's experience is optimal. By combining the total value of the user's permissions and the delay compensation predicted by the model, the audio and video delay tolerance threshold is dynamically adjusted. In the case of high-authority users, the delay prediction is smaller, and the system sets a lower delay tolerance threshold; while for low-authority users, the tolerance is increased accordingly; this enables the system to be flexibly adjusted according to the needs of different users, avoiding unnecessary delay corrections and improving overall efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a conference room audio and video synchronization visualization control method of the present invention; Figure 2 A flowchart of a hierarchical compensation algorithm for a conference room audio and video synchronization visualization control method of the present invention; Figure 3 This is a framework diagram of a conference room audio and video synchronization visualization control system of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0019] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0020] Example 1 Figure 1 A flowchart of a conference room audio and video synchronization visualization control method disclosed in this embodiment is shown, and the steps are as follows: S1. Extract the conference type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the permission opening value of all users through the dynamic weight strategy; S2. Monitor the device status of each conference member and obtain the device response impact value (the worse the device status, the more likely the member will face network delays and a higher compensation algorithm should be allocated) S3. A compensation algorithm is assigned to each participant through a hierarchical compensation algorithm, combined with the user permission opening value and device status; and the optimal parameter combination is searched through Bayesian optimization; S4. Set a synchronization correction strategy, dynamically adjust the audio and video delay tolerance threshold according to the permission opening value, and when the actual delay time is greater than the delay tolerance threshold, return to S3 and adjust the Bayesian optimization cycle of the user.

[0021] The dynamic weight strategy combines the three elements of role, behavior, and violation, and suppresses the expansion of permissions of long-term inactive users through logarithmic functions. The specific steps include: obtaining the user behavior score based on the behavior type of each participating user ; Obtain the cumulative value of the user violation event attenuation model through the violation events of each participating user ; And then get the permission value of each participating user ,in Indicates the user's basic role constant. ,in, On behalf of the visitor, Indicates members, Indicates the host, represents the time normalization base, Indicates the constant weight of the user's basic role, represents the user behavior score weight, Represents the cumulative value weight of the user violation event attenuation model.

[0022] The calculation process of the behavior score includes setting the speech duration interval, speech frequency interval and interaction times, and calculating the ratio of the historical average speech time of the participants of this type to the speech duration and the deviation time of the edge value to obtain the speech duration score. The speech frequency score is obtained in the same way. The speech duration score, speech frequency score and standardized interaction times of users with a time window of 30 days are taken as the behavior type and sent to the user behavior dataset. Get the user behavior score , where noa indicates the number of meetings of the participant within 30 days. Indicates that the participant The first Behavior type scores, Indicates The weight coefficient of each behavior type, represents the attenuation coefficient, Indicates The time when each behavior type occurs, Indicates the current time.

[0023] For example, if the speaking time interval is [5min, 10min], if the participant's speaking time is 3min, then the user behavior score of the participant is a / (5-3); if the participant's speaking time is 15min, then the user behavior score of the participant is a / (15-10), where a represents the historical average speaking time of this type of participant.

[0024] The cumulative value of the user violation event attenuation model includes determining user violation behavior. When the number of user interruptions is greater than 0 and less than the set first interruption threshold, it is determined to be a minor violation. When the number of user interruptions is greater than the set first interruption threshold, it is determined to be a severe violation. The violation behavior of the participating users with a time window of 30 days is extracted to obtain the user violation event attenuation model. , Indicates the severity of the defined violation event, H=720min indicates the half-decay period, Indicates The time when the violation occurred.

[0025] The device response impact value is obtained by monitoring the average value of the historical device synchronization delay of the conference members, and calculating the first Delay per meeting: ; in, Indicates Meeting No. The time of audio transmission of each speech, Indicates Meeting No. The time of the speech video, Indicates the number of user meetings within a 30-day time window. and Indicates the weight of the setting.

[0026] By monitoring the device response impact value, we can comprehensively consider the device performance and avoid calculation anomalies caused by network delays caused by occasional freezes, thereby wasting delay compensation resources.

[0027] Example 2 Based on the technical solution of embodiment 1, the present invention proposes a layered compensation algorithm: Figure 2 The flowchart of the hierarchical compensation algorithm of the conference room audio and video synchronization visualization control method disclosed in this embodiment is shown, and the steps are as follows: S31, obtaining the total authority value Tpe of each conference participant by weighted summing up the device response impact value and the authority opening value; S32, setting a first permission threshold TPe1 and a second permission threshold TPe2, when When When It is considered a high-authority scenario. In the present invention, high-authority users often speak frequently, and the length and importance of their speeches are relatively high. In order to ensure the experience of all meeting participants, the algorithm for high-authority users should be more complex. Low-authority users speak intermittently and for a short time, so a simpler compensation algorithm can also meet the needs of participants.

[0028] S33. A hierarchical compensation algorithm is used to realize the delay compensation requirements of different conference participants.

[0029] The layered compensation algorithm includes a basic delay compensation layer, an enhanced delay compensation layer and an advanced delay compensation layer; the basic delay compensation layer calculates the change of the playback rate ,in is the time difference between audio and video, is the frame time of the video, and the delay compensation of low authority is realized by changing the pitch without changing the speed; the enhanced delay compensation layer dynamically adjusts the buffer depth according to the total value of the authority, and the enhanced buffer depth , Indicates the size of the basic buffer; the advanced delay compensation layer is based on the LSTM network, dynamically adjusts the delay step and the signal path weight through the total authority value, and generates a synchronization signal.

[0030] The advanced delay compensation layer takes the Mel spectrum of the past M frames and the normalized permission value PTpe as input to dynamically adjust the model's predicted delay compensation ; The role of the authority value is to affect the adjustment of the delay step, ensuring that users with higher authority can obtain more accurate synchronization. The total authority value is used to assign weights to the signal paths. The weight of each path is expressed as , represents the quality score of the k-th path, Ensure that higher privilege users get better path weighting.

[0031] The audio and video delay tolerance threshold is calculated by compensating the delay predicted in step S3 As a basis, calculate the actual audio and video delay tolerance threshold , is a coefficient used to adjust the sensitivity of the tolerance threshold; Indicates the maximum audio and video delay deviation allowed under the current total authority value. By using the delay compensation value predicted by the model as the dynamic delay tolerance threshold, the system tolerance can be adjusted according to the predicted delay. For example, in the case of a high-authority user, the predicted delay may be small, so a smaller tolerance threshold can be set; in the case of a low-authority user, the delay prediction is large, and the tolerance threshold can be increased accordingly. When the absolute value of the difference between the actual arrival time of the audio and video stream and the expected time When the delay is greater than the audio and video delay tolerance threshold, the synchronization correction strategy is triggered, and the synchronization correction strategy includes setting a reference correction period , and obtain the updated correction period as the Bayesian optimization period .

[0032] The threshold and weight may be set by default according to the present invention, or may be set by an operator.

[0033] Example 3 Figure 3 The framework diagram of a conference room audio and video synchronization visualization control system of the present invention is shown. Based on the same inventive concept as Embodiment 1 and Embodiment 2, the present invention provides a conference room audio and video synchronization visualization control system, including: The user rights opening calculation module is used to extract the conference type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the rights opening values ​​of all users through the dynamic weight strategy; The device response monitoring module is used to monitor the device status of each conference member and obtain the device response impact value; The hierarchical compensation algorithm module is used to assign a compensation algorithm to each participant based on the user permission opening value and device status; and search for the optimal parameter combination through Bayesian optimization; The synchronization correction strategy module is used to dynamically adjust the audio and video delay tolerance threshold according to the permission opening value. When the actual delay time is greater than the delay tolerance threshold, it returns to S3 and adjusts the Bayesian optimization cycle of the user.

[0034] The synchronization correction strategy module includes an audio and video delay tolerance threshold calculation unit and an optimization cycle calculation unit; the audio and video delay tolerance threshold calculation unit is used to calculate the actual audio and video delay tolerance threshold according to the predicted delay compensation; the optimization cycle calculation unit is used to set the benchmark correction cycle and obtain the updated correction cycle as the Bayesian optimization cycle It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0035] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0036] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0037] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0038] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A conference room audio and video synchronization visualization control method, characterized in that: The following steps are involved: S1. Extract the conference type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the permission opening value of all users through the dynamic weight strategy; S2. Monitor the device status of each conference member and obtain the device response impact value; S3. A compensation algorithm is assigned to each participant through a hierarchical compensation algorithm, combined with the user permission opening value and device status; and the optimal parameter combination is searched through Bayesian optimization; S4. Set a synchronization correction strategy, dynamically adjust the audio and video delay tolerance threshold according to the permission opening value, and when the actual delay time is greater than the delay tolerance threshold, return to S3 and adjust the Bayesian optimization cycle of the user.

2. A conference room audio and video synchronization visualization control method according to claim 1, characterized in that: The dynamic weight strategy combines the three elements of role, behavior, and violation, and suppresses the expansion of the rights of long-term inactive users through a logarithmic function. The specific steps include: obtaining a user behavior score based on the behavior type of each participating user ; Obtain the cumulative value of the user violation event attenuation model through the violation events of each participating user ; And then get the permission value of each participating user ,in Indicates the user's basic role constant. ,in, On behalf of the visitor, Indicates members, Indicates the host, represents the time normalization base, Indicates the constant weight of the user's basic role, represents the user behavior score weight, Represents the cumulative value weight of the user violation event attenuation model.

3. A conference room audio and video synchronization visualization control method according to claim 1, characterized in that: The specific steps of S3 include: S31, obtaining the total authority value Tpe of each conference participant by weighted summing up the device response impact value and the authority opening value; S32, setting a first permission threshold TPe1 and a second permission threshold TPe2, when When When It is considered a high-authority scenario. S33. A hierarchical compensation algorithm is used to realize the delay compensation requirements of different conference participants.

4. A conference room audio and video synchronization visualization control method according to claim 3, characterized in that: The layered compensation algorithm includes a basic delay compensation layer, an enhanced delay compensation layer and an advanced delay compensation layer; the basic delay compensation layer calculates the change of the playback rate ,in is the time difference between audio and video, is the frame time of the video, and the delay compensation of low authority is realized by changing the pitch without changing the speed; the enhanced delay compensation layer dynamically adjusts the buffer depth according to the total value of the authority, and the enhanced buffer depth , Indicates the size of the basic buffer; the advanced delay compensation layer is based on the LSTM network, dynamically adjusts the delay step and the signal path weight through the total authority value, and generates a synchronization signal.

5. A conference room audio and video synchronization visualization control method according to claim 4, characterized in that: The advanced delay compensation layer takes the Mel spectrum of the past M frames and the normalized permission value PTpe as input to dynamically adjust the model's predicted delay compensation .

6. A conference room audio and video synchronization visualization control method according to claim 1, characterized in that: The audio and video delay tolerance threshold is calculated by compensating the delay predicted in step S3 As a basis, calculate the actual audio and video delay tolerance threshold , is a coefficient used to adjust the sensitivity of the tolerance threshold; when the absolute value of the difference between the actual arrival time of the audio and video stream and the expected time When the delay is greater than the audio and video delay tolerance threshold, the synchronization correction strategy is triggered, and the synchronization correction strategy includes setting a reference correction period , get the updated correction period .

7. A conference room audio and video synchronization visualization control method according to claim 2, characterized in that: The calculation process of the behavior score includes setting the speech duration interval, speech frequency interval and interaction times, and calculating the ratio of the historical average speech time of the participants of this type to the speech duration and the deviation time of the edge value to obtain the speech duration score. The speech frequency score is obtained in the same way. The speech duration score, speech frequency score and standardized interaction times of users with a time window of 30 days are taken as the behavior type and sent to the user behavior dataset. Get the user behavior score , where noa indicates the number of meetings of the participant within 30 days. Indicates that the participant The first participant Behavior type scores, Indicates The weight coefficient of each behavior type, represents the attenuation coefficient, Indicates The time when each behavior type occurs, Indicates the current time.

8. A conference room audio and video synchronization visualization control method according to claim 2, characterized in that: The cumulative value of the user violation event attenuation model includes determining user violation behavior. When the number of user interruptions is greater than 0 and less than the set first interruption threshold, it is determined to be a minor violation. When the number of user interruptions is greater than the set first interruption threshold, it is determined to be a severe violation. The violation behavior of the participating users with a time window of 30 days is extracted to obtain the user violation event attenuation model. , Indicates the severity of the defined violation event, H=720min indicates the half-decay period, Indicates The time when the violation occurred.

9. A conference room audio and video synchronization visualization control method according to claim 1, characterized in that: The device response impact value is obtained by monitoring the average value of the historical device synchronization delay of the conference members, and calculating the first Delay per meeting: ； in, Indicates Meeting No. The time of audio transmission of each speech, Indicates Meeting No. The time of the speech video, Indicates the number of user meetings within a 30-day time window. and Indicates the weight of the setting.

10. A conference room audio and video synchronization visualization control system, used to execute a conference room audio and video synchronization visualization control method according to any one of claims 1 to 9, characterized in that: include: The user rights opening calculation module is used to extract the meeting type of the conference room initiator, record the historical behavior data of the participating users in the current conference type, and calculate the rights opening values ​​of all users through the dynamic weight strategy; The device response monitoring module is used to monitor the device status of each conference member and obtain the device response impact value; The hierarchical compensation algorithm module is used to assign a compensation algorithm to each participant based on the user permission opening value and device status; and search for the optimal parameter combination through Bayesian optimization; The synchronization correction strategy module is used to dynamically adjust the audio and video delay tolerance threshold according to the permission opening value. When the actual delay time is greater than the delay tolerance threshold, it returns to S3 and adjusts the Bayesian optimization cycle of the user.

Images