A multi-person video conference picture layout method based on audio analysis

Abstract

The application discloses a multi-person video conference picture layout method based on audio analysis, which comprises the following steps: S1, assigning initial values of audio of each participant, namely, participant weight, speaking weight and window weight; S2, calculating speech energy of each audio; S3, calculating real-time speech coefficients of each audio; S4, updating the participant weight of each audio; S5, updating the window weight of each audio; S6, updating the main speaker; S7, updating the size coefficient of each video picture; S8, updating the position coefficient of each video picture; S9, calculating the actual size and position of each picture, namely, obtaining the final picture layout; the application can clearly and intuitively show the situation of multiple speakers in the conference, so that the participants can better focus on the speaker situation in the conference.

Description

Technical Field

[0001] This invention relates to the field of digital audio technology, and in particular to a method for layout of screens in multi-person video conferencing based on audio analysis. Background Technology

[0002] Currently, in conference room environments, conference tablets are often used for online meetings. A conference tablet is a smart tablet that integrates functions such as intelligent writing, wireless screen sharing, video conferencing, and remote collaboration. Conference tablets can install video conferencing software to conduct video conferences with other online users. In existing technical solutions, the screen layout is mainly as follows: 1. Based on who is currently speaking, the current speaker is placed on the main screen, and other non-speaking users are displayed as small images on the edge; 2. All participants are displayed on a separate screen, and the current speaker is identified by drawing a box outside the screen.

[0003] However, when there are many participants, the layout described above cannot clearly and intuitively display the most recent speakers in a multi-person discussion scenario. In complex discussion environments, especially when multiple participants speak almost simultaneously, this simple labeling method may not be enough to quickly attract the attention of all participants, thus missing key information. It also lacks the flexibility to dynamically adjust the screen according to the actual meeting and cannot effectively support diverse forms of meeting interaction.

[0004] Therefore, a novel method for layouting the screen in multi-person video conferencing is needed. This method can clearly and intuitively display the status of multiple speakers in the meeting, allowing participants to better focus on the speakers. Summary of the Invention

[0005] This invention provides a method for layout of screens in multi-person video conferencing based on audio analysis, comprising the following steps:

[0006] S1. Assign initial values ​​to the audio of each participant: participation weight, speaking weight, and window weight;

[0007] S2. Calculate the speech energy of each audio stream;

[0008] S3. Calculate the real-time speech coefficients of each audio stream by analyzing the speech energy sequence of each audio stream over a certain period of time.

[0009] S4. Update the participation weight of each audio stream based on the current voice coefficient of each audio stream in the meeting room.

[0010] S5. Update the window weight of each audio stream based on the current voice coefficient of each audio stream in the meeting room.

[0011] S6. Update the main speaker based on the current audio participation weights and window weights.

[0012] S7. Based on the current audio participation weights, voice coefficients, and window weights, update the size coefficients of each video frame;

[0013] S8. Update the position coefficients of each video frame based on the current weights of each audio window.

[0014] S9. Based on the current main speaker, the size coefficient of each screen, and the sorting coefficient of each screen, calculate the actual size and position of each screen to obtain the final screen layout.

[0015] Furthermore, speech energy characterizes the instantaneous intensity of sound energy in a frame of audio signal within an audio stream;

[0016] The audio PCM time domain uses data x[i];

[0017] Where x[i] is the time-domain amplitude sequence;

[0018] Let the current audio energy be Pv, then:

[0019] Pv=SUM(x[i]^2);

[0020] Here, i takes values ​​in the closed interval [a, b].

[0021] Furthermore, speech coefficients characterize the intensity of a speaker's speech over a very recent period of time in the audio stream;

[0022] Let the audio frame interval be T;

[0023] Let Pv[i] be the distance from the current time T. * The speech energy at time i, i.e.:

[0024] The current latest speech energy is Pv[0];

[0025] The speech energy of the previous frame is Pv[1];

[0026] The speech energy of the first two frames is Pv[2], and so on;

[0027] Let the current speech coefficient be Vp, then:

[0028] Vp=SUM(Pv[i] * (1 / (1+i / 5)));

[0029] i is to make T * All values ​​for which i < 1 second is true.

[0030] Furthermore, the participation weight represents the voice weight of one audio stream in the meeting up to the present.

[0031] Let there be N audio streams in the venue at the current moment, and let Vp[k] be the speech coefficient of the k-th audio stream.

[0032] For the k-th audio channel:

[0033] Let M[k][i] be the weight of the participants at the current frame time.

[0034] Let M[k][i-1] be the weight of the participants in the previous frame.

[0035] Let the initial participation weight be M[k][0]=1;

[0036] Let Vps be the sum of the voice coefficients of all N audio channels, then:

[0037] Vps = SUM(Vp[k]), where k belongs to [0, N];

[0038] Let A[k] be the participation gain of the k-th audio channel at the current moment, and let:

[0039] Am[k]=(Vp[k] / Vps>30%)?Vp[k]:0;

[0040] The current participation weight is calculated based on the participation gain at the current moment and the participation weight in the previous frame:

[0041] M[k][i]=0.8 * M[k][i-1]+Am[k].

[0042] Furthermore, the window weights are a sequence;

[0043] Let the audio frame interval be T;

[0044] The sequence length is L, let L = 60 seconds / T;

[0045] For the k-th audio channel, let the window weight be W[k], where the i-th element is W[k][i].

[0046] The value of i ranges from [0, L-1];

[0047] Let Vp[k][i] be the speech coefficient at time T*i from the current time, i.e.:

[0048] The latest speech coefficient is Vp[k][0];

[0049] The speech coefficient of the previous frame is Vp[k][1];

[0050] The speech coefficients for the first two frames are Vp[k][2], and so on;

[0051] Then, let W[k][i] = Vp[k][i].

[0052] Furthermore, regarding the speaker coefficient and speaker updates:

[0053] For the k-th audio channel, let the speaker coefficient of the current frame be S[k];

[0054] Let the audio frame interval be T;

[0055] Let S[k]=3 * M[k]+SUM(W[k][j]);

[0056] SUM is the summation of W[k][j] over a range of j, where j takes values ​​in the range [0, 15 seconds / T].

[0057] M[k] represents the k-th audio stream and the participation weight at the current moment;

[0058] Compare the main speaker coefficients of each user, and the user with the highest coefficient is the current main speaker.

[0059] Furthermore, regarding the update of the image size factor:

[0060] The size factor determines whether each image will be displayed, and at what size;

[0061] Let M[k][i] be the weight of the participants at the current frame time.

[0062] For the k-th audio path, let Q[k][i] be the size factor at the current frame time, and let Q[k][i-1] be the size factor at the previous frame time;

[0063] Let the initial size factor be Q[k][0] = 0;

[0064] Let W[k] be the sum of all elements of the weight of the audio window for the k-th audio channel;

[0065] The sum of W[k] of all audio signals is Ws;

[0066] If W[k] of the k-th audio path is not in the top five, then let Q[k][i] = 0;

[0067] If the W[k] / Ws of the k-th audio channel is less than 10%, then let Q[k][i]=0;

[0068] Other cases:

[0069] If Q[k][i-1]=0, then Q[k][i]=100;

[0070] Otherwise, Q[k][i] = Q[k][i-1] *(1+W[k] / Ws).

[0071] Furthermore, regarding the update of the image sorting coefficients:

[0072] The sorting coefficient determines the rules by which the various images are sorted.

[0073] Let k be the number of the participant whose current size factor is not 0;

[0074] Calculate the sorting coefficient P[k] for the k-th audio channel;

[0075] The window weight of the current audio path is W[k], where the value of each element is W[k][j].

[0076] The sorting coefficient P[k] = Q[k] > 0? M[k] + SUM(W[k][j]): 0;

[0077] Q[k] is the size coefficient of the k-th audio channel at the current moment;

[0078] SUM is the summation of W[k][j] over a range of j, where j takes values ​​from [0, 30 seconds / T].

[0079] M[k] represents the participation weight of the k-th audio stream at the current moment.

[0080] Furthermore, regarding image size calculation:

[0081] Let the total width of the current screen be WIDTH and the total height be HEIGHT;

[0082] For users with serial number k, the size factor is Q[k] and the sorting factor is P[k].

[0083] Let m be the current speaker's number, Qs be the sum of all users' size coefficients, and r be the number of users other than the speaker whose size coefficient is not 0.

[0084] The display width of the screen for user number k is width[k].

[0085] If k == m, then width[k] = (0.565 - 0.65) * r) * WIDTH;

[0086] If k != m, then width[k] = (0.435 - 0.65) * r) * WIDTH * Q[k] / (Qs–Q[m]);

[0087] The display height of the screen for user number k is height[k].

[0088] height[k]=(k==m||Q[k]>0)?HEIGHT:0

[0089] Furthermore, regarding the determination of the image position:

[0090] For users with serial numbers k, their sorting coefficient is P[k];

[0091] The speaker's image is placed on the far left;

[0092] For other users whose width[k]>0, they are arranged from left to right, starting from the right side of the main speaker, according to P[k] from largest to smallest.

[0093] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0094] This invention dynamically adjusts the video screen size and position of each participant by analyzing audio data in real time. This ensures that important speakers receive a larger display area, improving meeting efficiency and viewing experience. For participants who speak infrequently or less frequently, the system can reduce their screen size or even hide their screen, thus optimizing the overall layout of the video screen without sacrificing key information. Real-time adjustment of the screen layout can instantly reflect the dynamic changes in the meeting, allowing participants to more intuitively understand the meeting's progress and key points, enhancing interactivity and participation. Automated and intelligent screen layout adjustment reduces the burden on meeting organizers to manually adjust the screen, improving the efficiency and convenience of meeting management. Attached Figure Description

[0095] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0096] Figure 1 This is a schematic diagram of a multi-person video conferencing screen layout method based on audio analysis. Detailed Implementation

[0097] The technical solution of the present invention will now be described with reference to the accompanying drawings. However, the described embodiments are only some embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0098] like Figure 1 As shown, this invention provides a method for multi-person video conferencing screen layout based on audio analysis, including the following steps:

[0099] S1. Assign initial values ​​to the audio of each participant: participation weight, speaking weight, and window weight;

[0100] S2. Calculate the speech energy of each audio stream;

[0101] S3. Calculate the real-time speech coefficients of each audio stream by analyzing the speech energy sequence of each audio stream over a certain period of time.

[0102] S4. Update the participation weight of each audio stream based on the current voice coefficient of each audio stream in the meeting room.

[0103] S5. Update the window weight of each audio stream based on the current voice coefficient of each audio stream in the meeting room.

[0104] S6. Update the main speaker based on the current audio participation weights and window weights.

[0105] S7. Based on the current audio participation weights, voice coefficients, and window weights, update the size coefficients of each video frame;

[0106] S8. Update the position coefficients of each video frame based on the current weights of each audio window.

[0107] S9. Based on the current main speaker, the size coefficient of each screen, and the sorting coefficient of each screen, calculate the actual size and position of each screen to obtain the final screen layout.

[0108] In this embodiment, speech energy represents the instantaneous intensity of sound energy in a frame of audio signal in an audio stream;

[0109] The audio PCM time domain uses data x[i];

[0110] Where x[i] is the time-domain amplitude sequence;

[0111] Let the current audio energy be Pv, then:

[0112] Pv=SUM(x[i]^2);

[0113] Here, i takes values ​​in the closed interval [a, b].

[0114] In this embodiment, the speech coefficient represents the speaking intensity of the speaker in the audio stream over a very recent period of time;

[0115] Let the audio frame interval be T;

[0116] Let Pv[i] be the distance from the current time T. * The speech energy at time i, i.e.:

[0117] The current latest speech energy is Pv[0];

[0118] The speech energy of the previous frame is Pv[1];

[0119] The speech energy of the first two frames is Pv[2], and so on;

[0120] Let the current speech coefficient be Vp, then:

[0121] Vp=SUM(Pv[i] * (1 / (1+i / 5)));

[0122] i is to make T * All values ​​for which i < 1 second is true.

[0123] In this embodiment, the participation weight represents the voice weight of one audio stream in the meeting up to the present.

[0124] Let there be N audio streams in the venue at the current moment, and let Vp[k] be the speech coefficient of the k-th audio stream.

[0125] For the k-th audio channel:

[0126] Let M[k][i] be the weight of the participants at the current frame time.

[0127] Let M[k][i-1] be the weight of the participants in the previous frame.

[0128] Let the initial participation weight be M[k][0]=1;

[0129] Let Vps be the sum of the voice coefficients of all N audio channels, then:

[0130] Vps = SUM(Vp[k]), where k belongs to [0, N];

[0131] Let Am[k] be the participation gain of the k-th audio channel at the current moment, and let:

[0132] Am[k]=(Vp[k] / Vps>30%)?Vp[k]:0;

[0133] The current participation weight is calculated based on the participation gain at the current moment and the participation weight in the previous frame:

[0134] M[k][i]=0.8 * M[k][i-1]+Am[k].

[0135] In this embodiment, the window weights are a sequence;

[0136] Let the audio frame interval be T;

[0137] The sequence length is L, let L = 60 seconds / T;

[0138] For the k-th audio channel, let the window weight be W[k], where the i-th element is W[k][i].

[0139] The value of i ranges from [0, L-1];

[0140] Let Vp[k][i] be the distance from the current time T for the k-th audio path. * The speech coefficients at time i, i.e.:

[0141] The latest speech coefficient is Vp[k][0];

[0142] The speech coefficient of the previous frame is Vp[k][1];

[0143] The speech coefficients for the first two frames are Vp[k][2], and so on;

[0144] Then, let W[k][i] = Vp[k][i].

[0145] In this embodiment, the speaker coefficient and speaker update are as follows:

[0146] For the k-th audio channel, let the speaker coefficient of the current frame be S[k];

[0147] Let the audio frame interval be T;

[0148] Let S[k]=3 * M[k]+SUM(W[k][j]);

[0149] SUM is the summation of W[k][j] over a range of j, where j takes values ​​in the range [0, 15 seconds / T].

[0150] M[k] represents the k-th audio stream and the participation weight at the current moment;

[0151] Compare the main speaker coefficients of each user, and the user with the highest coefficient is the current main speaker.

[0152] In this embodiment, the image size factor is updated as follows:

[0153] The size factor determines whether each image will be displayed, and at what size;

[0154] Let M[k][i] be the weight of the participants at the current frame time.

[0155] For the k-th audio path, let Q[k][i] be the size factor at the current frame time, and let Q[k][i-1] be the size factor at the previous frame time;

[0156] Let the initial size factor be Q[k][0] = 0;

[0157] Let W[k] be the sum of all elements of the weight of the audio window for the k-th audio channel;

[0158] The sum of W[k] of all audio signals is Ws;

[0159] If W[k] of the k-th audio path is not in the top five, then let Q[k][i] = 0;

[0160] If the W[k] / Ws of the k-th audio channel is less than 10%, then let Q[k][i]=0;

[0161] Other cases:

[0162] If Q[k][i-1]=0, then Q[k][i]=100;

[0163] Otherwise, Q[k][i] = Q[k][i-1] * (1+W[k] / Ws).

[0164] In this embodiment, the image sorting coefficient is updated as follows:

[0165] The sorting coefficient determines the rules by which the various images are sorted.

[0166] Let k be the number of the participant whose current size factor is not 0;

[0167] Calculate the sorting coefficient P[k] for the k-th audio channel;

[0168] The window weight of the current audio path is W[k], where the value of each element is W[k][j].

[0169] The sorting coefficient P[k] = Q[k] > 0? M[k] + SUM(W[k][j]): 0;

[0170] Q[k] is the size coefficient of the k-th audio channel at the current moment;

[0171] SUM is the summation of W[k][j] over a range of j, where j takes values ​​from [0, 30 seconds / T].

[0172] M[k] represents the participation weight of the k-th audio stream at the current moment.

[0173] In this embodiment, the calculation of screen size is as follows:

[0174] Let the total width of the current screen be WIDTH and the total height be HEIGHT;

[0175] For users with serial number k, the size factor is Q[k] and the sorting factor is P[k].

[0176] Let m be the current speaker's number, Qs be the sum of all users' size coefficients, and r be the number of users other than the speaker whose size coefficient is not 0.

[0177] The display width of the screen for user number k is width[k].

[0178] If k == m, then width[k] = (0.565 - 0.65) * r) * WIDTH;

[0179] If k != m, then width[k] = (0.435 - 0.65) * r) * WIDTH * Q[k] / (Qs–Q[m]);

[0180] The display height of the screen for user number k is height[k].

[0181] height[k]=(k==m||Q[k]>0)?HEIGHT:0

[0182] In this embodiment, the image position is determined as follows:

[0183] For users with serial numbers k, their sorting coefficient is P[k];

[0184] The speaker's image is placed on the far left;

[0185] For other users whose width[k]>0, they are arranged from left to right, starting from the right of the main speaker, according to their P[k] values ​​from largest to smallest. That is, the user with the largest P[k] is next to the main speaker on the left and on the far left of the other users, while the user with the smallest P[k] value is on the far right of the screen.

[0186] This invention has many applications, including but not limited to the following described scenarios:

[0187] In remote work scenarios, multi-person video conferencing has become an important way of daily communication. This invention can automatically optimize the screen layout to ensure that the screens of important speakers are highlighted, thereby improving meeting efficiency.

[0188] In online education scenarios, teachers may need to interact with students frequently. This invention can automatically adjust the screen layout between teachers and students based on the speaking situation to ensure that students can clearly see the teacher's demonstration and explanation.

[0189] In telemedicine consultations, communication between doctors and patients is crucial. By automatically adjusting the screen layout, key statements from either the doctor or the patient can be highlighted, thus improving the effectiveness of the consultation.

[0190] In corporate training, instructors may need to interact and discuss frequently with trainees. This invention can adjust the screen layout in real time to ensure that the screens of instructors and trainees are displayed reasonably, thereby improving the training effect.

[0191] In large online seminars, manually adjusting the screen layout becomes impractical due to the large number of participants. This invention can automatically adjust the screen layout according to the speaking situation, ensuring that the screens of important speakers are highlighted, while optimizing the overall screen layout and improving the viewing experience for participants.

Claims

1. A method for layout of screens in a multi-person video conferencing system based on audio analysis, characterized in that, Includes the following steps: S1. Assign initial values ​​to the audio of each participant: participation weight, speaking weight, and window weight; S2. Calculate the speech energy of each audio stream; where speech energy represents the instantaneous intensity of sound energy in a frame of audio signal in the audio stream. The audio PCM time domain uses data x[i]; Where x[i] is the time-domain amplitude sequence; Let the current audio energy be Pv, then: Pv = SUM(x[i]^2); Where i takes values ​​in the closed interval [a, b]; S3. Calculate the real-time speech coefficient of each audio stream by taking the speech energy sequence of each audio stream over a certain period of time; where the speech coefficient represents the speaking intensity of the speaker in the audio stream over a very short period of time. Let the audio frame interval be T; Let Pv[i] be the distance from the current time T. * The speech energy at time i, i.e.: The current latest speech energy is Pv[0]; The speech energy of the previous frame is Pv[1]; The speech energy of the first two frames is Pv[2], and so on; Let the current speech coefficient be Vp, then: Vp = SUM( Pv[i] * (1 / (1+i / 5)) ) ; i is to make T * All values ​​for which i < 1 second is true; S4. Update the participation weight of each audio stream based on the current voice coefficient of each audio stream in the meeting room. Among them, the participation weight represents the voice weight of one audio stream in the meeting up to the present; Let there be N audio streams in the venue at the current moment, and let Vp[k] be the speech coefficient of the k-th audio stream. For the k-th audio channel: Let M[k][i] be the weight of the participants at the current frame time. Let M[k][i-1] be the weight of the participants in the previous frame. Let the initial participation weight be M[k][0]=1; Let Vps be the sum of the voice coefficients of all N audio channels, then: Vps = SUM(Vp[k]), where k belongs to [0,N]; Let Am[k] be the participation gain of the k-th audio channel at the current moment, and let: Am[k] = (Vp[k] / Vps > 30%) ? Vp[k] : 0; The current participation weight is calculated based on the participation gain at the current moment and the participation weight in the previous frame: M[k][i] = 0.8 * M[k][i-1] + Am[k]; S5. Update the window weight of each audio stream based on the current speech coefficient of each audio stream in the meeting room; where the window weight is a sequence. Let the audio frame interval be T; The sequence length is L, let L = 60 seconds / T; For the k-th audio channel, let the window weight be W[k], where the i-th element is W[k][i]. The value of i ranges from [0, L-1]; Let Vp[k][i] be the distance from the current time T for the k-th audio path. * The speech coefficients at time i, i.e.: The latest speech coefficient is Vp[k][0]; The speech coefficient of the previous frame is Vp[k][1]; The speech coefficients for the first two frames are Vp[k][2], and so on; Then, let W[k][i] = Vp[k][i]; S6. Based on the current audio participant weights and window weights, update the main speaker; specifically, regarding the main speaker coefficient and the main speaker update: For the k-th audio channel, let the speaker coefficient of the current frame be S[k]; Let the audio frame interval be T; Let S[k] = 3 * M[k]+SUM(W[k][j]); SUM is the summation of W[k][j] over a range of j, where j ranges from [0, 15 seconds / T]. M[k] represents the k-th audio stream and the participation weight at the current moment; Compare the main speaker coefficients of each user, and the user with the highest coefficient is the current main speaker; S7. Based on the current audio participant weights, voice coefficients, and window weights, update the size coefficients of each video frame; specifically, for the frame size coefficient update: The size factor determines whether each image will be displayed, and at what size; Let M[k][i] be the weight of the participants at the current frame time. For the k-th audio path, let Q[k][i] be the size factor at the current frame time, and let Q[k][i-1] be the size factor at the previous frame time; Let the initial size factor be Q[k][0] = 0; Let W[k] be the sum of all elements of the weight of the audio window for the k-th audio channel; The sum of W[k] of all audio signals is Ws; If W[k] of the k-th audio path is not in the top five, then let Q[k][i] = 0; If the W[k] / Ws of the k-th audio channel is less than 10%, then let Q[k][i]=0; Other cases: If Q[k][i-1]=0, then Q[k][i]=100; Otherwise, Q[k][i] = Q[k][i-1] * (1 + W[k] / Ws); S8. Update the position coefficients of each video frame based on the current weights of each audio window. S9. Based on the current main speaker, the size coefficient of each screen, and the sorting coefficient of each screen, calculate the actual size and position of each screen to obtain the final screen layout.

2. The method for multi-person video conferencing screen layout based on audio analysis according to claim 1, characterized in that, For updating the image sorting coefficients: The sorting coefficient determines the rules by which the various images are sorted. Let k be the number of the participant whose current size factor is not 0; Calculate the sorting coefficient P[k] for the k-th audio channel; The window weight of the current audio path is W[k], where the value of each element is W[k][j]. The sorting coefficient P[k] = Q[k] > 0 ? M[k] + SUM(W[k][j]) : 0; Q[k] is the size coefficient of the k-th audio channel at the current moment; SUM is the summation of W[k][j] over a range of j, where j ranges from [0, 30 seconds / T]. M[k] represents the participation weight of the k-th audio stream at the current moment.

3. The method for multi-person video conferencing screen layout based on audio analysis according to claim 2, characterized in that, For image size calculation: Let the total width of the current screen be WIDTH and the total height be HEIGHT; For users with serial number k, the size factor is Q[k] and the sorting factor is P[k]. Let m be the current speaker's number, Qs be the sum of all users' size coefficients, and r be the number of users other than the speaker whose size coefficient is not 0. The display width of the screen for user number k is width[k]. If k == m, then width[k] = (0.565 - 0.65) * r) * WIDTH; If k != m, then width[k] = (0.435 - 0.65 * r) * WIDTH * Q[k] / (Qs – Q[m]); The display height of the screen for user number k is height[k]. height[k] = (k == m || Q[k] > 0) ? HEIGHT :

0.

4. The method for multi-person video conferencing screen layout based on audio analysis according to claim 3, characterized in that, Regarding the determination of the image position: For users with serial numbers k, their sorting coefficient is P[k]; The speaker's image is placed on the far left; For other users whose width[k] > 0, they are arranged from left to right, starting from the right side of the main speaker, according to P[k] from largest to smallest.

Images