Large-screen video conference window adjusting method based on image processing

Abstract

The invention discloses a large-screen video conference window adjusting method based on image processing, which relates to the technical field of image processing, and comprises the following steps: preparation before conference, screen division, image processing, detection of personnel image data in a main window and an auxiliary window by using a video quality detection algorithm in a video conference to obtain a window quality score Qvid, according to the method, a video window is divided into a main window and a plurality of auxiliary windows through a video window division algorithm, image enhancement processing and speech adjustment are carried out on the window, the priority score Spring of each window is obtained through a spokesman detection algorithm, the highest score is judged as a spokesman and a conference record, the video window is divided into a main window and a plurality of auxiliary windows through a video window division algorithm, and the current spokesman can be obviously seen; according to the method, the priority score Spri is obtained through a spokesman detection algorithm, the highest score is judged as the spokesman, the highest score is automatically set as the spokesman, manual adjustment is not needed, and therefore the current spokesman can be intelligently and effectively determined, and the experience effect of a large-screen video conference is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a large-screen video conference window adjustment method based on image processing. Background Art

[0002] A video conference is a real-time visual connection between two or more remote parties, which simulates a face-to-face meeting. Image stitching refers to combining multiple video streams or image sources in a video conference into a single large-screen display, which can increase interaction and communication between participants and improve the efficiency and quality of the meeting. Image stitching technology can combine these video streams into a whole picture to provide a better viewing experience.

[0003] In current video conferences, all participants' images are displayed on the screen. When speaking, it is difficult for participants to notice the specific speaker, which makes the conference process more chaotic. When there are many people speaking during the conference, it is necessary to constantly manually switch the display screen of the video conference client, which makes the switching process of the display screen of the video conference client complicated and unable to intelligently and effectively determine the current speaker, resulting in a poor experience of large-screen video conferencing and reduced conference quality. Summary of the invention

[0004] The purpose of the present invention is to provide a large-screen video conference window adjustment method based on image processing, which improves the quality of the conference and solves the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a large-screen video conference window adjustment method based on image processing, comprising the following steps:

[0006] Preparation before the meeting: build a large-screen display system, which includes a display screen, obtain image data of people participating in the video conference, and project it on the video window of the display screen;

[0007] Screen division: Use the video window division algorithm to divide the video window into a main window and several sub-windows. The main window is used to display the speakers participating in the video conference. The main window occupies a larger display area than the sub-windows. The sum of the main windows and sub-windows is equal to the number of people participating in the video conference.

[0008] Image processing: In video conferencing, the video quality detection algorithm is used to detect the image data of people in the main window and the sub-window to obtain the window quality score Q vid , and set the window quality score Q vid The lowest threshold is Y1, when the window quality score Q vid When it is less than Y1, the window is subjected to image enhancement processing;

[0009] Speech adjustment: During a large-screen video conference, the speaker detection algorithm is used to derive the priority score S for each window. pri The person with the highest score is determined as the speaker, and its image data is replaced with the image data on the main window, and the priority score S is set. pri The quality threshold is Y2, when the priority score S pri When it is less than Y2, the data in the video quality detection algorithm is adjusted;

[0010] Meeting records: Use recording software to record the video conference process and obtain meeting data. After the large-screen video conference is over, the meeting data is stored and tagged for subsequent reference.

[0011] Optionally, the large-screen display system includes a display screen, a signal processing and transmission device, a control device and an audio device.

[0012] Optionally, the main window size calculation process in the video window division algorithm is as follows:

[0013] W m =R m ×W,H m =R m ×H

[0014] Where W m is the main window width, H m is the height of the main window, m is the main window;

[0015] R m It is the ratio of the main window to the video window, and the value range is between 0 and 1;

[0016] W is the total width of the video window, and H is the total height of the video window;

[0017] The secondary window size calculation process is as follows:

[0018]

[0019] Where W s is the width of the secondary window, H s is the height of the secondary window, s is the secondary window;

[0020] N is the total number of people in the video conference;

[0021] N-1 is the number of sub-windows;

[0022] is the number of columns of the sub-window;

[0023] The calculation process of the sub-window arrangement position is as follows:

[0024]

[0025] Where X si is the horizontal coordinate of the i-th window;

[0026] Y si is the ordinate of the i-th window;

[0027] is the number of rows in the secondary window;

[0028] The video window division algorithm can automatically generate a suitable conference window layout based on the number of people participating in the video conference, making the layout of the video windows on the display screen more neat and the main window can highlight the speaker.

[0029] Optionally, the video quality detection algorithm process in the image processing step is as follows:

[0030]

[0031] Where Q vid (i) is the quality score of the i-th window, Q vid is the window quality score;

[0032] W vidi ×H vidi is the resolution of the ith window, indicating the clarity of the person image data;

[0033] W vidi is the width of the i-th window video, W max Is the maximum width of the video;

[0034] H vidi is the height of the i-th window video; H max is the maximum height of the video;

[0035] F vidi is the video frame rate of the ith window, F mzx is the maximum frame rate of the video;

[0036] K r is the resolution weight, the value is 0.5;

[0037] K f is the frame rate weight, the value is 0.5;

[0038] The i-th window quality score Q vid (i) represents the clarity and smoothness of the person image data, setting the window quality score Q vid The lowest threshold is Y1, when the i-th window quality score Q vid When (i) is less than Y1, it means that the video in the window is blurred and stuck, and the overall video quality is low. At this time, image enhancement processing is performed on the video in the window.

[0039] Optionally, the speaker detection algorithm process in the speech adjustment step is as follows:

[0040] S pri (i) = P spe (i) × Q vid (i)

[0041] Where S pri (i) is the priority score of the i-th window, S pri is the priority score;

[0042] S spe (i) is the speaking probability of the i-th window

[0043] Q vid (i) is the quality score of the i-th window;

[0044] Specifically, S. pri (i) The process of obtaining is as follows:

[0045] S pri (i) = S spe,face (i) × β + S spe,audio (i)×γ

[0046] Where S spe,face (i) is the probability of facial speech based on the i-th window;

[0047] S spe,audio (i) is the probability of the i-th window based on the audio speech;

[0048] β is the facial speech probability weight, ranging from 0.5 to 1;

[0049] γ is the audio speech probability weight, ranging from 0.5 to 1;

[0050] Specific:

[0051]

[0052] Where S spe,face (i) is the probability of facial speech based on the i-th window;

[0053] f mouth,i is the mouth opening and closing frequency of the i-th window;

[0054] f spr It is the standard opening and closing frequency of the mouth when pronouncing a word;

[0055] α mouth,i is the opening and closing range of the mouth;

[0056] α max The maximum opening and closing range of the mouth;

[0057]

[0058] Where S spe,audio (i) is the probability of the i-th window based on the audio speech;

[0059] E signal,i is the audio energy value of the i-th window;

[0060] E th is the audio energy threshold;

[0061] Priority Score pri Indicates the probability of the window speaking, the priority score S pri The highest window will be replaced with the main window when the priority score S pri When the highest window is the main window, there will be no changes, ensuring the stability of the video conference.

[0062] Optionally, the setting priority score S pri The quality threshold is Y2. When the priority score of a window is S pri When it is less than Y2, it means that the quality of the current window image data is too low, and the window resolution weight K needs to be increased. r The proportion in the video quality detection algorithm is as follows:

[0063] K r =0.5+Z

[0064] When the window resolution weight K r In order to ensure K r +K f =1, then K f The corresponding reduction is as follows:

[0065] K f =0.5-Z

[0066] Z is the adjustment amount, ranging from 0.1 to 0.4;

[0067] Since the speaker detection algorithm needs to detect the face of the person, if the window resolution is too low, it will affect the accuracy of face detection. At this time, the priority score S of each window is used. pri , when the priority score S pri When it is too low, it can affect the data in the video quality detection algorithm, increase the weight of resolution, and make the window quality score Q vid Ability to make real-time adjustments to specific conditions of window quality.

[0068] Optionally, the image enhancement processing includes noise removal, super-resolution reconstruction and frame rate interpolation.

[0069] Optionally, the conference data includes video data and audio data, wherein the video data includes data before image processing and data after image processing.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. The present invention divides the video window into a main window and several sub-windows through a video window division algorithm. The main window is used to display the speaker, and the several sub-windows are neatly distributed in other positions, which can efficiently utilize the screen space and avoid overlap and waste. In addition, a suitable conference window layout can be generated according to the number of participants in the video conference. Regardless of the number of participants in the video conference, the interface remains neat and uncrowded, so that the participants in the video conference can clearly see the current speaker, which can effectively improve the concentration and improve the effect of the video conference.

[0072] Second, the present invention detects the personnel image data of each window through a video quality detection algorithm, and obtains the window quality score Q by combining the resolution and frame rate of the window. vid , set the window quality score Q vid Minimum threshold. When the window quality score is less than the minimum threshold, the video in the window is enhanced to ensure the accuracy of the video image in each window, making the participant more recognizable. Then, the speaker detection algorithm is used to derive the priority score S based on the facial and audio data in each window. pri The person with the highest score is determined as the speaker, and the software settings automatically replace his or her image data with the image data on the main window without manual adjustment. This can intelligently and effectively determine the current speaker, increase the experience of large-screen video conferencing, and improve the quality of the meeting.

[0073] 3. The present invention simultaneously scores the priority S pri Set the quality threshold, when the priority score S pri When it is less than the quality threshold, it means that the window image quality is too low and data cannot be detected from the window. Since the speaker detection algorithm needs to detect the frequency of the person's mouth opening and closing, if the window resolution is too low, it will affect the accuracy of the detection. At this time, the data in the video quality detection algorithm is adjusted to increase the weight of the resolution so that the window quality score Q vid Ability to make real-time adjustments to the specific conditions of window quality and improve window quality score Q vid The flexibility of the algorithm allows the algorithms to influence each other and jointly improve the effect of large-screen video conferencing. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] For examples, see Figure 1 , this implementation provides a large-screen video conference window adjustment method based on image processing, including the following steps:

[0077] Preparation before the meeting: build a large-screen display system, which includes a display screen, obtain image data of people participating in the video conference, and project it on the video window of the display screen;

[0078] Screen division: Use the video window division algorithm to divide the video window into a main window and several sub-windows. The main window is used to display the speakers participating in the video conference. The main window occupies a larger display area than the sub-windows. The sum of the main windows and sub-windows is equal to the number of people participating in the video conference.

[0079] Image processing: In video conferencing, the video quality detection algorithm is used to detect the image data of people in the main window and the sub-window to obtain the window quality score Q vid , and set the window quality score Q vid The lowest threshold is Y1, when the window quality score Q vid When it is less than Y1, the window is subjected to image enhancement processing;

[0080] Speech adjustment, speech adjustment, during the large-screen video conference, the speaker detection algorithm is used to obtain the priority score S of each window pri The person with the highest score is determined as the speaker, and its image data is replaced with the image data on the main window, and the priority score S is set. pri The quality threshold is Y2, when the priority score S pri When it is less than Y2, the data in the video quality detection algorithm is adjusted;

[0081] Meeting records: Use recording software to record the video conference process and obtain meeting data. After the large-screen video conference is over, the meeting data is stored and tagged for subsequent reference.

[0082] Specifically, the conference data includes video data and audio data, wherein the video data includes data before image processing and data after image processing. By dividing the video data into data before image processing and data after image processing, it is possible to clearly know the image enhancement effect during subsequent viewing, so as to decide whether adjustment and optimization are needed.

[0083] In this embodiment, the video window is divided into a main window and several sub-windows through a video window division algorithm. The main window is used to display the speaker. Since the main window is larger than the sub-window, the sub-windows are neatly distributed in other positions, which can efficiently utilize the screen space and avoid overlap and waste. In addition, the number of sub-windows can be dynamically adjusted as the number of participants in the video conference changes, ensuring that the interface remains neat and uncrowded regardless of the number of participants, so that other video conference participants can clearly see the current speaker, which can effectively improve concentration and improve the meeting effect. After the main window and the sub-window are divided, the personnel image data of each window is detected by the video quality detection algorithm, and the window quality score Q is obtained by combining the resolution and frame rate of the window. vid , window quality score Q vid The level of Q represents the clarity and smoothness of the window, and is set by vid The lowest threshold is Y1, when the window quality score Q vid When it is less than Y1, the window is enhanced. The video quality detection algorithm can automatically perform image enhancement on the window video with lower quality, ensuring the accuracy of each window video while reducing the consumption of computer performance, improving the accuracy of subsequent recognition, making the meeting process smoother, and enhancing the comfort of large-screen video conferencing.

[0084] Finally, the speaker detection algorithm is used to obtain the priority score S based on the facial and audio data in each window. pri The person with the highest score is determined as the speaker, and the software settings automatically replace the person's image data with the person's image data on the main window, without manual adjustment. This can intelligently and effectively determine the current speaker, increase the experience of large-screen video conferencing, improve the quality of the meeting, and at the same time give the priority score S pri Set the quality threshold to Y2, when the priority score S pri When it is less than Y2, it means that the window image quality is too low and data cannot be detected from the window. At this time, the data in the video quality detection algorithm is adjusted so that it can change the window quality score Q according to the actual situation. vid , improving the flexibility of this method.

[0085] Furthermore, the large-screen display system includes a display screen, a signal processing and transmission device, a control device and an audio device.

[0086] Specifically, the display screen is an LED display screen, which supports large-size display while ensuring high-resolution image quality. The LED display screen has good brightness and contrast, and can remain clearly visible under different lighting conditions. In multi-person video conferences, it can ensure that participants can clearly see the current speaker in a complex visual environment. The signal processing device is used to process audio and video signals collected from cameras, microphones and other devices to improve the quality of video conferences.

[0087] The control device is the command center of the entire large-screen display system. It is responsible for coordinating the source of the video signal, adjusting the layout, switching the video window, etc. The control device starts, pauses or adjusts the video signal output of different participants according to the needs of the system, and automatically or manually switches the speaker window to ensure the smooth progress of the meeting.

[0088] Furthermore, the main window size calculation process in the video window division algorithm is as follows:

[0089] W m =R m ×W,H m =R m ×H

[0090] Where W m is the main window width, H m is the height of the main window, m is the main window;

[0091] R m It is the ratio of the main window to the video window, and the value range is between 0 and 1;

[0092] W is the total width of the video window, and H is the total height of the video window;

[0093] The secondary window size calculation process is as follows:

[0094]

[0095] Where W s is the width of the secondary window, H s is the height of the secondary window, s is the secondary window;

[0096] N is the total number of people in the video conference;

[0097] N-1 is the number of sub-windows;

[0098] is the number of columns of the sub-window;

[0099] The calculation process of the sub-window arrangement position is as follows:

[0100]

[0101] Where Xsi is the horizontal coordinate of the i-th window;

[0102] Y si is the ordinate of the i-th window;

[0103] is the number of rows in the secondary window;

[0104] The video window division algorithm can divide the video window on the display screen into a main window and several sub-windows, and can generate a suitable conference window layout according to the number of people participating in the video conference. In addition, the number of sub-windows can be dynamically adjusted as the number of participants in the video conference changes. Regardless of the number of participants in the video conference, the interface remains neat and uncrowded, so that the video conference participants can clearly see the current speaker, which can effectively improve concentration and enhance the effectiveness of the video conference.

[0105] Furthermore, the video quality detection algorithm process in the image processing step is as follows:

[0106]

[0107] Where Q vid (i) is the quality score of the i-th window, Q vid is the window quality score;

[0108] W vidi ×H vidi is the resolution of the ith window, indicating the clarity of the person image data;

[0109] W vidi is the width of the i-th window video, W max Is the maximum width of the video;

[0110] H vidi is the height of the i-th window video; H max is the maximum height of the video;

[0111] F vidi is the video frame rate of the ith window, F mzx is the maximum frame rate of the video;

[0112] K r is the resolution weight, the value is 0.5;

[0113] K f is the frame rate weight, the value is 0.5;

[0114] Specifically, the i-th window quality score Q vid (i) represents the clarity and smoothness of the person image data, setting the window quality score Q vid The lowest threshold is Y1, when the i-th window quality score Qvid When (i) is less than Y1, it means that the video in the window is blurred and stuck, and the overall video quality is low. At this time, the video in the window is enhanced. The video quality detection algorithm can automatically perform image enhancement processing on the window with too low video quality to ensure the accuracy of the video image in each window, so that the participant recognition is higher and the accuracy of the subsequent speaker detection algorithm is improved.

[0115] Furthermore, the speaker detection algorithm process in the speech adjustment step is as follows:

[0116] S pri (i) = P spe (i) × Q vid (i)

[0117] Where S pri (i) is the priority score of the i-th window, S pri is the priority score;

[0118] S spe (i) is the speaking probability of the i-th window

[0119] Q vid (i) is the quality score of the i-th window;

[0120] Specifically, S. pri (i) The process of obtaining is as follows:

[0121] S pri (i) = S spe,face (i) × β + S spe,audio (i)×γ

[0122] Where S spe,face (i) is the probability of facial speech based on the i-th window;

[0123] S spe,audio (i) is the probability of the i-th window based on the audio speech;

[0124] β is the facial speech probability weight, ranging from 0.5 to 1;

[0125] γ is the audio speech probability weight, ranging from 0.5 to 1;

[0126] Specific:

[0127]

[0128] Where S spe,face (i) is the probability of facial speech based on the i-th window;

[0129] f mouth,i is the mouth opening and closing frequency of the i-th window;

[0130] f spr It is the standard opening and closing frequency of the mouth when pronouncing a word;

[0131] α mouth,i is the opening and closing range of the mouth;

[0132] α max The maximum opening and closing range of the mouth;

[0133]

[0134] Where S spe,audio (i) is the probability of the i-th window based on the audio speech;

[0135] E signal,i is the audio energy value of the i-th window;

[0136] E th is the audio energy threshold;

[0137] Specifically, the priority score S pri It indicates the speaking probability of the window. Through the information of facial speaking probability and audio speaking probability, the system can more accurately determine whether the current window is in the speaking state. For example, the audio energy value in the audio speaking probability can effectively identify whether the pronunciation exists, and the mouth opening and closing frequency further confirms whether there is a corresponding mouth movement or expression that matches the audio energy value. The two cooperate with each other. Even in an environment with background noise, this method can improve the recognition accuracy through the complementary characteristics of multimodality and avoid misjudging non-speakers or other participants as speakers. The priority score S pri The highest window will be replaced with the main window when the priority score S pri When the highest is the main window, there will be no change, ensuring the stability of the video conference without manual adjustment. This can intelligently and effectively determine the current speaker, increase the experience of large-screen video conferencing, and improve the quality of the meeting.

[0138] Furthermore, we set the priority score S pri The quality threshold is Y2. When the priority score of a window is S pri When it is less than Y2, it means that the quality of the current window image data is too low, and the window resolution weight K needs to be increased. r The proportion in the video quality detection algorithm is as follows:

[0139] K r =0.5+Z

[0140] When the window resolution weight K r In order to ensure K r +K f =1, then Kf The corresponding reduction is as follows:

[0141] K f =0.5-Z

[0142] Z is the adjustment amount, ranging from 0.1 to 0.4;

[0143] Since the speaker detection algorithm needs to detect the frequency of a person's mouth opening and closing, if the window resolution is too low, it will affect the accuracy of the detection. In this case, the priority score S of each window is used. pri , when the priority score S pri When it is too low, it can affect the data in the video quality detection algorithm, increase the weight of resolution, and make the window quality score Q vid Ability to make real-time adjustments to the specific conditions of window quality and improve window quality score Q vid flexibility.

[0144] Furthermore, image enhancement processing includes noise removal, super-resolution reconstruction and frame rate interpolation.

[0145] Specifically, noise removal is used to remove noise components introduced by shooting equipment, ambient light, network transmission, etc. from video frames. Noise usually appears as random points, spots or detail loss in the image, which seriously affects the visual quality. In video conferencing, noise removal can reduce interference on the screen, obtain clearer and more natural images, and enhance the viewing experience of conference participants. Super-resolution reconstruction can increase the number of pixels and details of the image, avoid distortion, and more accurately capture people's facial features. Frame rate interpolation increases the video frame rate by inserting intermediate frames. New frames are inserted through algorithms to smooth the transition between the original frames, avoid freezes or discontinuities during video playback, and improve the fluency of video conferencing.

[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A large-screen video conference window adjustment method based on image processing, characterized in that: The following steps are involved: Step S1: Pre-conference preparation, build a large-screen display system, the large-screen display system includes a display screen, obtain image data of people participating in the video conference, and project it on the video window of the display screen; Step S2: screen division: using a video window division algorithm to divide the video window into a main window and a plurality of sub-windows, wherein the main window is used to display the speakers participating in the video conference, the main window occupies a larger display screen area than the sub-windows, and the sum of the number of main windows and sub-windows is equal to the number of people participating in the video conference; Step S3: Image processing: Use the video quality detection algorithm in the video conference to detect the image data of the people in the main window and the sub-window to obtain the window quality score Q vid , and set the window quality score Q vid The lowest threshold is Y1, when the window quality score Q vid When it is less than Y1, the window is subjected to image enhancement processing; Step S4: Speech adjustment: During the large-screen video conference, the speaker detection algorithm is used to obtain the priority score S of each window. pri The person with the highest score is determined as the speaker, and its image data is replaced with the image data on the main window, and the priority score S is set. pri The quality threshold is Y2, when the priority score S pri When it is less than Y2, the data in the video quality detection algorithm is adjusted; Step S5: Meeting record, use recording software to record the video conference process to obtain meeting data. After the large-screen video conference is over, the meeting data is stored and tagged for subsequent reference.

2. The large-screen video conference window adjustment method based on image processing according to claim 1 is characterized in that: The large-screen display system includes a display screen, a signal processing and transmission device, a control device and an audio device.

3. The large-screen video conference window adjustment method based on image processing according to claim 2 is characterized in that: The main window size calculation process in the video window division algorithm is as follows: W m =R m ×W,H m =R m ×H Where W m is the main window width, H m is the height of the main window, m is the main window; R m It is the ratio of the main window to the video window, and the value range is between 0 and 1; W is the total width of the video window, and H is the total height of the video window; The secondary window size calculation process is as follows: Where W s is the width of the secondary window, H s is the height of the secondary window, s is the secondary window; N is the total number of people in the video conference; N-1 is the number of sub-windows; is the number of columns of the sub-window; The calculation process of the sub-window arrangement position is as follows: Where X si is the horizontal coordinate of the i-th window; Y si is the ordinate of the i-th window; is the number of rows in the secondary window; The video window division algorithm can automatically generate a suitable conference window layout based on the number of people participating in the video conference, making the layout of the video windows on the display screen more neat and the main window can highlight the speaker.

4. The large-screen video conference window adjustment method based on image processing according to claim 3 is characterized in that: The video quality detection algorithm process in the image processing step is as follows: Where Q vid (i) is the quality score of the i-th window, Q vid is the window quality score; W vidi ×H vidi is the resolution of the ith window, indicating the clarity of the person image data; W vidi is the width of the i-th window video, W max Is the maximum width of the video; H vidi is the height of the i-th window video; H max is the maximum height of the video; F vidi is the video frame rate of the ith window, F mzx is the maximum frame rate of the video; K r is the resolution weight, the value is 0.5; K f is the frame rate weight, the value is 0.5; The i-th window quality score Q vid (i) represents the clarity and smoothness of the person image data, setting the window quality score Q vid The lowest threshold is Y1, when the i-th window quality score Q vid When (i) is less than Y1, it means that the video in the window is blurred and stuck, and the overall video quality is low. At this time, image enhancement processing is performed on the video in the window.

5. The large-screen video conference window adjustment method based on image processing according to claim 4 is characterized in that: The speaker detection algorithm process in the speech adjustment step is as follows: S pri (i)=P spe (i)×Q vid (i) Where S pri (i) is the priority score of the i-th window, S pri is the priority score; S spe (i) is the speaking probability of the i-th window Q vid (i) is the quality score of the i-th window; Specifically, S. pri (i) The process of obtaining is as follows: S pri (i)=S spe,face (i)×β+S spe,audio (i)×γ Where S spe,face (i) is the probability of facial speech based on the i-th window; S spe,audio (i) is the probability of the i-th window based on the audio speech; β is the facial speech probability weight, ranging from 0.5 to 1; γ is the audio speech probability weight, ranging from 0.5 to 1; Specific: Where S spe,face (i) is the probability of facial speech based on the i-th window; f mouth,i is the mouth opening and closing frequency of the i-th window; f spr It is the standard opening and closing frequency of the mouth when pronouncing a word; α mouth,i is the opening and closing range of the mouth; α max The maximum opening and closing range of the mouth; Where S spe,audio (i) is the probability of the i-th window based on the audio speech; E signal,i is the audio energy value of the i-th window; E th is the audio energy threshold; Priority Score pri Indicates the probability of the window speaking, the priority score S pri The highest window will be replaced with the main window when the priority score S pri When the highest window is the main window, there will be no changes, ensuring the stability of the video conference.

6. The large-screen video conference window adjustment method based on image processing according to claim 5 is characterized in that: Set the priority score S pri The quality threshold is Y2. When the priority score of a window is S pri When it is less than Y2, it means that the quality of the current window image data is too low, and the window resolution weight K needs to be increased. r The proportion in the video quality detection algorithm is as follows: K r =0.5+Z When the window resolution weight K r In order to ensure K r +K f =1, then K f The corresponding reduction is as follows: K f =0.5-Z Z is the adjustment amount, ranging from 0.1 to 0.4; Since the speaker detection algorithm needs to detect the face of the person, if the window resolution is too low, it will affect the accuracy of face detection. At this time, the priority score S of each window is used. pri , when the priority score S pri When it is too low, it can affect the data in the video quality detection algorithm, increase the weight of resolution, and make the window quality score Q vid Ability to make real-time adjustments to specific conditions of window quality.

7. The large-screen video conference window adjustment method based on image processing according to claim 4 is characterized in that: The image enhancement process includes noise removal, super-resolution reconstruction and frame rate interpolation.

8. The large-screen video conference window adjustment method based on image processing according to claim 2 is characterized in that: The conference data includes video data and audio data, wherein the video data includes data before image processing and data after image processing.

Images