A method, terminal and storage medium for collaborative video conferencing

Abstract

The present invention provides a method, a terminal and a storage medium for collaborative video conferencing. The method includes: obtaining voice information respectively input by multiple terminal devices; selecting the terminal device with the highest decibel among the multiple voice information as the target terminal device; closing the voice input of other terminal devices except the target terminal device, deploying multiple video conferencing terminal devices according to the size of the meeting room. When initializing and deploying the video conferencing terminal devices, the video conferencing terminal devices in the same meeting room are assigned to the same meeting group. When having a meeting, the video conferencing terminal devices in the same group send the collected video data to the local video conferencing software, and after performing picture splicing processing on the video data, it is sent to the video conferencing server to facilitate other participants to understand the current meeting situation, which can achieve the best audio-visual effect in the same meeting room and better video conferencing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of video conferencing, and particularly to a method, a terminal and a storage medium for collaborative video conferencing among multiple parties. Background Art

[0002] The multi-party video conferencing technology is a multimedia communication technology that enables people in different locations to achieve "real-time, visual, and interactive" video communication through a certain transmission medium. It can transmit various information such as static and dynamic images, voices, and texts of participants to the terminal devices of other participants through various existing communication transmission media, enabling participants scattered geographically to exchange information in multiple ways such as video, image, and sound, and making users feel as if they are participating in the same meeting on the spot.

[0003] Generally, in a multi-party video conferencing system, the server collects the videos sent by each client, then integrates the videos sent by all clients in the meeting, and sends them to each participating client, thus realizing a multi-person video conference.

[0004] However, currently, generally only one video conferencing terminal is placed in a meeting room in a multi-party video conferencing system. But when there is only one video conferencing terminal, the sound pickup effect of the participants who are far away from the meeting terminal is relatively poor, and the situation of dragging the meeting terminal often occurs. And if multiple meeting terminals are deployed, there will be situations of repeated pictures and chaotic sound crossovers, and the following two problems often occur:

[0005] 1. Frequently dragging the meeting terminal is time-consuming and laborious, and when dragging the meeting terminal, it will generate relatively large noise and picture shaking, which will affect the listening and viewing feelings of other participating parties.

[0006] 2. Replacing the meeting terminal device with better effects will increase the cost, and the viewing angle of a single meeting terminal is ultimately limited, and it is impossible to include all participants in the picture, resulting in the situation where one can only hear their voices but not see their faces. Summary of the Invention

[0007] In order to solve the above problems, the present invention provides a method, a terminal and a storage medium for collaborative video conferencing among multiple parties.

[0008] The present invention is realized through the following technical solutions:

[0009] The present invention provides a method for collaborative video conferencing among multiple parties, and the method includes:

[0010] Obtaining voice information respectively input by multiple terminal devices;

[0011] Selecting the terminal device with the largest decibel among the multiple voice information as the target terminal device;

[0012] Turn off the voice input of other terminal devices except the target terminal device.

[0013] Further, before obtaining the voice information input by multiple terminal devices, it also includes:

[0014] Obtain the physical address of each terminal and generate coding information according to the physical address;

[0015] Specifically, obtaining the voice information input by multiple terminal devices includes:

[0016] Make the obtained voice information correspond one by one with the coding information.

[0017] Further, after turning off the voice input of other terminal devices except the target terminal device, it also includes:

[0018] Record the number of times of speaking when different terminal devices are used as the target terminal device;

[0019] Record the number of speakers when different terminal devices are used as the target terminal device;

[0020] Record the speaking duration when different terminal devices are used as the target terminal device.

[0021] Further, after recording the speaking duration when different terminal devices are used as the target terminal device, it also includes:

[0022] Analyze the obtained data and assign weights to multiple terminal devices.

[0023] Further, before obtaining the voice information input by multiple terminal devices, it also includes:

[0024] Judge whether the terminal device has received voice information;

[0025] If not, repeatedly execute the judgment on whether the terminal device has received voice information;

[0026] If so, obtain the decibel value of the voice information.

[0027] Further, recording the number of times of speaking when different terminal devices are used as the target terminal device includes:

[0028] Judge whether there is any other terminal device inputting voice information;

[0029] If not, increment the number of times of speaking of the terminal device by one;

[0030] If so, the number of times of speaking of the terminal device remains unchanged.

[0031] Further, recording the number of speakers when different terminal devices are used as the target terminal device includes:

[0032] Determine whether it is the same person speaking;

[0033] If not, increment the number of speakers of the terminal device by one;

[0034] If so, the number of speakers of the terminal device remains unchanged.

[0035] Further, recording the speaking duration when different terminal devices are used as the target terminal device includes:

[0036] Determine whether the speech of the same person has ended;

[0037] If not, record the cumulative speaking time of the terminal device;

[0038] If so, continue to accumulate the speaking time of the terminal device.

[0039] A terminal, the terminal includes: a memory, a processor, and a multi-video conference collaborative meeting program stored on the memory and running on the processor. When the multi-video conference collaborative meeting program is executed by the processor, it implements the multi-video conference collaborative meeting method according to any one of claims 1 to 8.

[0040] A storage medium, the storage medium stores a multi-video conference collaborative meeting program. When the multi-video conference collaborative meeting program is executed by a processor, it implements the multi-video conference collaborative meeting program according to any one of claims 1 to 8.

[0041] The beneficial effects of the present invention:

[0042] The multi-video conference collaborative meeting method, terminal, and storage medium proposed by the present invention provide an implementation device and method for multi-video conference terminal collaborative meetings. Deploy multiple video conference terminal devices according to the size of the meeting room. When initializing and deploying the video conference terminal devices, the video conference terminals in the same meeting room are assigned to the same meeting group. When having a meeting, the video conference terminal devices in the same group send the collected video data to the local video conference software, and after performing picture splicing processing on the video data, send it to the video conference server. The video conference server then sends it to the video conference software of other participants. Among the meeting terminals in the same group, only one meeting terminal can play sound at the same time, and the meeting terminal that needs to play sound can be dynamically selected according to the amount of speech detected by each meeting terminal. In terms of sound pickup, all video conference terminal devices in the same group perform sound pickup by selecting the nearest video conference terminal device based on sound source localization. Whether to support simultaneous speech can be selected through configuration in the same group. If simultaneous speech is not supported, when a certain video conference terminal detects speech, the microphones of other meeting terminals in the group are automatically turned off. According to the sound source direction of the meeting terminal that detects speech, the local conference software is notified to mark the region of interest in the video picture and notify other participants, facilitating other participants to understand the current meeting situation. Through the above solution, the best audio and video effects can be achieved in the same meeting room, and the video conference efficiency is better, with the following three advantages:

[0043] 1. Improve the meeting efficiency of video conferences.

[0044] 2. Improve the audio and video quality in video conferences.

[0045] 3. Improve the sense of participation of participants in video conferences. Description of the Drawings

[0046] Figure 1 Schematic diagram of the single meeting room deployment networking solution of the present invention;

[0047] Figure 2 Schematic diagram of the multi-party meeting application networking solution of the present invention;

[0048] Figure 3 Flowchart of the video conference terminal device joining and status service of the present invention;

[0049] Figure 4 Flowchart of the video conference software selecting the video conference terminal as the sound pickup device of the present invention;

[0050] Figure 5 Flowchart of the video conference software selecting the loudspeaker device of the present invention;

[0051] Figure 6Schematic diagram of the internal environment of the terminal of the present invention. Detailed implementation manners

[0052] To more clearly and completely illustrate the technical solution of the present invention, the present invention will be further described below with reference to the accompanying drawings.

[0053] Please refer to Figures 1 to 6 , the present invention provides a method, a terminal and a storage medium for collaborative video conferencing. The present invention provides a method for collaborative video conferencing, and the method includes:

[0054] Obtain voice information respectively input by a plurality of terminal devices;

[0055] Select the terminal device with the highest decibel among the plurality of voice information as the target terminal device;

[0056] Turn off the voice input of other terminal devices except the target terminal device.

[0057] Further, before obtaining the voice information respectively input by a plurality of terminal devices, it further includes:

[0058] Obtain the physical address of each terminal and generate coding information according to the physical address;

[0059] Specifically, obtaining the voice information respectively input by a plurality of terminal devices includes:

[0060] Make the obtained voice information correspond to the coding information one by one.

[0061] Further, after turning off the voice input of other terminal devices except the target terminal device, it further includes:

[0062] Record the number of speaking times when different terminal devices are used as the target terminal device;

[0063] Record the number of speakers when different terminal devices are used as the target terminal device;

[0064] Record the speaking duration when different terminal devices are used as the target terminal device.

[0065] Further, after recording the speaking duration when different terminal devices are used as the target terminal device, it further includes:

[0066] Analyze the obtained data and assign weights to a plurality of terminal devices.

[0067] Further, before obtaining the voice information respectively input by a plurality of terminal devices, it further includes:

[0068] Judge whether the terminal device receives voice information;

[0069] If not, repeatedly execute the step of judging whether the terminal device receives voice information;

[0070] If so, obtain the decibel value of the voice information.

[0071] Further, recording the number of speaking times of different terminal devices as the target terminal device includes:

[0072] Determine whether there is any other terminal device inputting voice information;

[0073] If not, increment the number of speaking times of the terminal device by one;

[0074] If so, the number of speaking times of the terminal device remains unchanged.

[0075] Further, recording the number of speakers of different terminal devices as the target terminal device includes:

[0076] Determine whether it is the same person speaking;

[0077] If not, increment the number of speakers of the terminal device by one;

[0078] If so, the number of speakers of the terminal device remains unchanged.

[0079] Further, recording the speaking duration of different terminal devices as the target terminal device includes:

[0080] Determine whether the speech of the same person has ended;

[0081] If not, record the cumulative speaking time of the terminal device;

[0082] If so, continue to accumulate the speaking time of the terminal device.

[0083] In this embodiment, according to the size of the meeting room, deploy a corresponding number of video conferencing terminal devices. For the convenience of explanation in this solution, 3 video terminal devices are used. Their numbers are MT-A, MT-B, and MT-C respectively. The following elaborates on the specific solution details.

[0084] In this solution, in the meeting room, in addition to configuring video conferencing terminal devices, a display and a host need to be configured. For example, a meeting tablet computer (a device integrating the host and the display) can be used, or a conventional computer host and a display (two separate devices of the host and the display) can be used. The host needs to install corresponding meeting software that can process audio and video to achieve the purpose of multi-party video conferencing.

[0085] In order to achieve the purpose of multi-party video conferencing, this solution needs to be divided into the following 4 stages. They are the implementation and deployment stage, the meeting preparation stage, the meeting in-progress stage, and the meeting end stage. The following separately describes the specific tasks completed in each stage.

[0086] I. Deployment Phase

[0087] 1. Connect the video conferencing terminal device to the video conferencing host via a USB cable. The video conferencing terminal device is connected via the USB Video Class / USB Audio Class protocol. When the video conferencing host detects the connected video conferencing terminal, it will assign device numbers respectively. Assume the numbering rule is MT-X, where X starts from the capital letter A and goes up to Z, and can be extended according to the actual situation. For example, the number of the first video conferencing terminal device connected to the video conferencing host is MT-A, the second is MT-B, the third is MT-C, and so on.

[0088] 2. After the video conferencing terminal device is connected to the video conferencing host, through the USB device descriptor, in addition to reporting the device name and device version, it also needs to report the MAC address of the device to uniquely identify the device.

[0089] 3. When the video conferencing software runs, it automatically detects the existing video conferencing terminal devices and stores them in the database according to the generated numbers. If the original existing device (judged according to the MAC address) does not exist in this detection, it is set to the offline state. Among the devices detected this time, if the original device already exists, it is set to the online state. If the device is a new device, it is written into the database for storage and set to the online state. The storage status of the devices detected this time in the database is as follows:

[0090] ID Device Number Device Name Device MAC Device Type Status 1 MT-A MeettingTerminal-A 11-22-33-44-55-66 Audio, Video Online 2 MT-B MeettingTerminal-B 22-33-44-55-66-77 Video Online 3 MT-C MeettingTerminal-C 33-44-55-66-77-88 Audio, Video Online 4 MT-D MeettingTerminal-D 44-55-66-77-88-99 Audio Offline

[0091] 4. When the video conferencing software runs, it generates a unique UUID according to the hardware information such as the MAC address and CPU ID of the device where the software runs. All video conferencing terminal devices connected to the current video conferencing host are automatically divided into the group corresponding to the current UUID.

[0092] II. Preparation Phase

[0093] 1. When the video conferencing software is connected to the cloud video conferencing server, multi-party video conferencing can be carried out.

[0094] 2. When preparing to start a multi-party meeting, the video conferencing software reports the grouping information of the current video conferencing software to the video conferencing server. At the same time, the local video conferencing software captures the images of the devices that support video capture functions according to the devices in the current grouping information and performs scene stitching. The specific scene stitching technology is an existing technology and will not be elaborated in detail in this solution.

[0095] 3. The video conferencing server associates the reported grouping information of each conference room with the created conference ID for storage, facilitating subsequent tracking of participant information and understanding of the conference progress. The grouping information of each conference room only includes the video conferencing terminal devices participating in the current conference. Offline or non-participating video conferencing terminal devices will not be reported to the video conferencing server.

[0096] 4. When the grouping ID already exists on the video conferencing server, the original grouping information is automatically updated; otherwise, new grouping information is added.

[0097] 5. After the video conferencing server establishes a connection with the conference host, the conference can enter the session stage. Subsequently, for the participant information invited by the conference host to join the current conference, the grouping information is automatically associated with the corresponding conference ID. And the joining and leaving times of each participant are recorded.

[0098] 6. The information related to the current conference stored on the video conferencing server is as follows:

[0099]

[0100]

[0101] 7. The grouping information stored on the video conferencing server is as follows:

[0102] Group ID Device MAC Device Type f46e5e53-00a3-4ad0-bef7-91d2695d7049 11-22-33-44-55-66 Audio, Video f46e5e53-00a3-4ad0-bef7-91d2695d7049 22-33-44-55-66-77 Video f46e5e53-00a3-4ad0-bef7-91d2695d7049 33-44-55-66-77-88 Audio, Video 0b7bd4a6-7129-4bef-91ea-b34ac27ff68a aa-bb-cc-dd-ee-ff Audio, Video 0b7bd4a6-7129-4bef-91ea-b34ac27ff68a bb-cc-dd-ee-ff-gg Audio 01aa2db1-c4a0-4d5f-a32b-de81b0666e3f 1a-2b-3c-4d-5e-6f Audio, Video 00e7dd8d-aeb4-4047-8109-3842ad560eda zz-yy-xx-ww-vv-uu Audio, Video

[0103] III. Session Stage

[0104] 1. When entering the conference session stage, when the video conferencing server detects that other participants (except the conference host) join the conference, it notifies the host and other participants to process the audio and video data and send it to the video conferencing server.

[0105] 2. The video conferencing server forwards the received video data to the video conferencing software of other participants (except the video source) for displaying the video and voice.

[0106] 3. The video information saved on the video conferencing server is as follows.

[0107]

[0108]

[0109] 4. The video forwarding information saved on the video conferencing server is as follows.

[0110] StreamID Destination bPKpxbcc7Z 0b7bd4a6-7129-4bef-91ea-b34ac27ff68a bPKpxbcc7Z 01aa2db1-c4a0-4d5f-a32b-de81b0666e3f bPKpxbcc7Z 00e7dd8d-aeb4-4047-8109-3842ad560eda Xj3muLZTVd f46e5e53-00a3-4ad0-bef7-91d2695d7049 Xj3muLZTVd 01aa2db1-c4a0-4d5f-a32b-de81b0666e3f Xj3muLZTVd 00e7dd8d-aeb4-4047-8109-3842ad560eda LRaQCsrDm8 f46e5e53-00a3-4ad0-bef7-91d2695d7049 LRaQCsrDm8 0b7bd4a6-7129-4bef-91ea-b34ac27ff68a LRaQCsrDm8 00e7dd8d-aeb4-4047-8109-3842ad560eda o8OFRSEwbw f46e5e53-00a3-4ad0-bef7-91d2695d7049 o8OFRSEwbw 0b7bd4a6-7129-4bef-91ea-b34ac27ff68a o8OFRSEwbw 01aa2db1-c4a0-4d5f-a32b-de81b0666e3f

[0111] 5. Initially, set the default video conferencing terminal as the sound input device and the sound output device. For example, set the first detected video conferencing terminal device as the sound input device and the sound output device, and turn off the sound pickup and speaker functions of other video conferencing terminal devices. During the meeting, if each video conferencing terminal detects sound input, then count the energy magnitude of the detected sound, expressed in dB. If multiple terminals detect sound, pick up the sound with the video conferencing terminal with the maximum energy, and mute the other microphones. Suppose there are currently three video conferencing terminal devices. Since only two devices support microphones, only two microphones have the sound pickup effect and can detect sound. The sound magnitudes detected by the two devices are MT-A 56dB and MT-C 48dB respectively. Then when the video terminal device transmits the detected sound energy magnitude to the video conferencing software, the video conferencing software will track the sound energy magnitude, select MT-A as the sound pickup microphone, and turn off the microphone of MT-C. At the same time, notify MT-C to turn off the sound pickup. And count the number of times each video conferencing terminal device serves as the sound pickup microphone and the number of microphones with different timbres. At the same moment, only one microphone can have the sound pickup effect, and other microphones are in the off state to avoid echo phenomena.

[0112] 6. When other microphones are turned off, ensure that one microphone can pick up sound normally. If the normally sound-picking-up microphone has abnormal sound pickup, it is necessary to promptly notify the video conferencing software to select and turn on other video conferencing terminal devices for sound pickup to prevent sound loss. This solution can also use multiple video conferencing terminal devices to pick up sound simultaneously, and on the video conferencing software, select the sound with the maximum energy according to the sound energy magnitude and transmit it to the video conferencing server. At the same time, the sound can be superimposed and eliminated to prevent echo effects.

[0113] 7. When the video conferencing terminal device with the main sound pickup detects a change in timbre (i.e., different speakers), it is necessary to promptly notify the video conferencing software. The video conferencing software notifies other video conferencing terminal devices to turn on the microphones for sound pickup in a timely manner, and repeat the processing process of step 5. At a certain moment, the information of each conference terminal stored in the video conferencing software is as follows:

[0114] Device number Device MAC Number of speeches Number of speakers Speech duration (minutes) MT-A 11-22-33-44-55-66 3 5 20 MT-C 33-44-55-66-77-88 1 3 32

[0115] 8. When the video conferencing software receives the audio and video data of other participants sent by the video conferencing server, the video data is decoded and then output to the display. The audio data is weighted and calculated or selected according to the strategy for the device to play the sound based on the number of speaking times and the number of speakers on each video conferencing terminal device recorded on the video conferencing software. Different strategy methods can be selected according to the strategy, such as giving priority to the number of speaking times, giving priority to the number of speakers, giving priority to the speaking duration, and weighted calculation priority. Among them, the weighted calculation method can calculate the final weight W = αNnumber of times + βNnumber of speakers + λTspeaking duration according to the calculation formula. The coefficients α, β, and λ can be set according to the actual situation, and α + β + λ = 1. Suppose the current selection is to give priority to the number of speakers strategy, then according to the currently stored data, the MT-A video conferencing terminal device should be selected as the speaker device to play the sound externally.

[0116] 9. Obtain the approximate azimuth information of the current speaker according to the voice algorithm - sound source localization algorithm. Send the azimuth information to the video conferencing software. When the video conferencing software performs image processing, it automatically marks the approximate area of the current speaker. On this basis, special image processing can be carried out. For example, automatically center the speaker's picture, perform digital zooming, etc., or combined with the camera form, the pan-tilt control can be performed to adjust the camera orientation, etc.

[0117] IV. Meeting end stage

[0118] 1. When a participant's video conferencing software exits the video conference, notify the video conferencing server. When the video conferencing server receives the exit message, it updates the current participant information, deletes the participant who exited the conference, and cancels sending the audio and video data of other conference parties to the participant who exited the conference.

[0119] 2. When the last participant exits the conference, the corresponding video conference on the video conferencing server ends, and the occupied resources are released, etc.

[0120] Based on the above multi-video-conference collaborative meeting method, the present invention also discloses a terminal 1, which includes: a memory 2, a processor 3, and a multi-video-conference collaborative meeting program 4 stored on the memory and running on the processor. When the multi-video-conference collaborative meeting program 4 is executed by the processor 3, the above multi-video-conference collaborative meeting method is implemented.

[0121] Based on the above multi-video-conference collaborative meeting method, the present invention also discloses a storage medium, which stores the multi-video-conference collaborative meeting program 4. When the multi-video-conference collaborative meeting program 4 is executed by the processor 3, the above multi-video-conference collaborative meeting method is implemented.

[0122] Of course, the present invention may have many other embodiments. Based on this embodiment, other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present invention.

Claims

1. A method for collaborative video conferencing, characterized in that, The method includes: Obtaining voice information respectively input by multiple terminal devices, selecting the terminal device with the highest decibel among the multiple voice information as the target terminal device serving as the pick-up microphone, and turning off the voice input of other terminal devices except the target terminal device serving as the pick-up microphone; When the target terminal device serving as the pick-up microphone detects a change in timbre, it notifies the video conferencing software, and the video conferencing software notifies other terminal devices to turn on the microphone for voice pickup, and repeats the previous step; Recording the number of speaking times of different terminal devices when they are the target terminal device, recording the number of speakers of different terminal devices when they are the target terminal device, and recording the speaking duration of different terminal devices when they are the target terminal device; assigning weights to the number of speaking times, the number of speakers, and the speaking duration, so as to select a terminal device as the speaker device for playing the audio data of other participants sent by the video conferencing server to play the sound externally.

2. The multi-video conference collaborative meeting method according to claim 1, wherein Before obtaining the voice information respectively input by multiple terminal devices, it further includes: Obtaining the physical address of each terminal and generating encoding information according to the physical address; Specifically obtaining the voice information respectively input by multiple terminal devices includes: Making the obtained voice information correspond one-to-one with the encoding information.

3. The multi-video conferencing collaborative meeting method according to claim 1, characterized in that, Before obtaining the voice information respectively input by multiple terminal devices, it further includes: Judging whether the terminal device has received voice information; If not, repeatedly execute the judgment on whether the terminal device has received voice information; If so, obtaining the decibel value of the voice information.

4. The multi-video conference collaborative meeting method according to claim 1, wherein Recording the number of speaking times of different terminal devices when they are the target terminal device includes: Judging whether there is any other terminal device inputting voice information; If not, adding 1 to the number of speaking times of the terminal device; If so, the number of speaking times of the terminal device remains unchanged.

5. The multi-video conference collaborative meeting method according to claim 1, characterized in that Recording the number of speakers of different terminal devices when they are the target terminal device includes: Judging whether it is the same person speaking; If not, adding 1 to the number of speakers of the terminal device; If so, the number of speakers of the terminal device remains unchanged.

6. The multi-video conference collaborative meeting method according to claim 1, wherein Recording the speaking duration of different terminal devices when they are the target terminal device includes: Judging whether the speaking of the same person has ended; If not, recording the cumulative speaking time of the terminal device; If so, continuing to accumulate the speaking time of the terminal device.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multi-video conferencing collaborative meeting program stored on the memory and running on the processor. When the multi-video conferencing collaborative meeting program is executed by the processor, it implements the multi-video conferencing collaborative meeting method according to any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium stores a multi-video conferencing collaborative meeting program. When the multi-video conferencing collaborative meeting program is executed by a processor, it implements the multi-video conferencing collaborative meeting method according to any one of claims 1 to 6.

Images