Virtual conference control method, device, computer readable storage medium and computer program product

Abstract

This application provides a virtual meeting control method, device, computer program product, and computer-readable storage medium; including determining the microphone status and voice signal of each participant in the virtual meeting, and determining the speaker in the virtual meeting based on the microphone status and voice signal; determining the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the view mode of the private virtual image acquisition device corresponding to each non-speaking participant; for the virtual image acquisition device to be moved, determining the target position of the virtual image acquisition device to be moved; the target position is a position that fully displays the upper body or full body of the speaker to be displayed; controlling the virtual image acquisition device to be moved to move in a straight line from the original position to the target position, and capturing and displaying the image of the speaker to be displayed at the moved position.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a virtual conference control method, device, computer-readable storage medium, and computer program product. Background Technology

[0002] In existing virtual meetings, participants typically need to manually move their virtual avatars to a designated location after entering the meeting room, and the viewpoint remains fixed. This means that when someone speaks, if the current viewpoint is unsuitable, participants can only switch to a more comfortable position by pressing buttons. However, this operation is complex, time-consuming, and may fail to achieve a suitable viewpoint even after multiple attempts, severely impacting the meeting experience. To address this, a solution for automatically adjusting the viewpoint has been proposed. When a second user is detected speaking, the system uses the coordinates of the second avatar corresponding to that user in the virtual meeting scenario, along with the coordinates of the first avatars corresponding to other first users (excluding the second user). The system uses the coordinates and current viewpoint of each first AI user to calculate viewpoint adjustment parameters. Based on these parameters, it adjusts the current viewpoint of each first AI user to switch their gaze towards the second AI user. It also adjusts the display screen of the corresponding first user based on their current gaze direction to show the second AI user. However, the related technology still has the following problems: the solution only focuses on a single person, making it impossible to observe all users simultaneously when multiple people are speaking. Furthermore, in situations with a large number of AI users and a large space, it cannot guarantee that the speaker can be seen clearly and completely after the viewpoint adjustment. Summary of the Invention

[0003] This application provides a virtual meeting control method, device, computer-readable storage medium, and computer program product, which solves the problems in related technologies that only focus on a single person, cannot observe all people at the same time when multiple people are speaking, and cannot guarantee that the speaker can be seen clearly and completely after the perspective is adjusted when there are many people and a large space.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a virtual conference control method, the method comprising: Determine the microphone status and voice signal of each participant in the virtual meeting, and determine the speaker in the virtual meeting based on the microphone status and the voice signal; Based on the number of speakers, the duration of each speaker's voice output, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant, the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device is determined; wherein, the perspective mode includes single-person mode and multi-person mode; For the virtual image acquisition device to be moved, determine the target position of the virtual image acquisition device to be moved; wherein, the target position is the position that fully displays the upper body or full body of the speaker to be displayed; the virtual meeting is also configured with at least one public virtual image acquisition device; the virtual image acquisition device to be moved includes: the private virtual image acquisition device corresponding to the non-speaking object among all participants and / or the public virtual image acquisition device used to display the secondary speaking group in the multi-person mode; The virtual image acquisition device is controlled to move from its original position to the target position along a straight line, and then captures and displays an image of the speaker at the new position.

[0005] In the above scheme, determining the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the viewpoint mode of the private virtual image acquisition device corresponding to each participant includes: If there are multiple speakers and the perspective mode is single-person mode, a target speaker is determined from the multiple speakers based on the voice output duration of each speaker; wherein, the speaker to be displayed includes the target speaker. The target speaker is determined to be displayed on each of the aforementioned private virtual image acquisition devices.

[0006] In the above scheme, determining the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the viewpoint mode of the private virtual image acquisition device corresponding to each participant includes: If the number of speakers is multiple and the perspective mode is a multi-person mode, all speakers are grouped to obtain at least one speaking group; wherein, the speakers to be displayed include the at least one speaking group; The total voice output duration of each speaking group is determined based on the voice output duration of each speaking object within each speaking group. The speaking group with the longest total voice output duration is determined as the main speaking group, and the other speaking groups besides the main speaking group are determined as secondary speaking groups. The screen display size for each speaking group is determined based on the total duration of multiple voice outputs. The main speaking group is displayed on each of the private virtual image acquisition devices, and each of the secondary speaking groups is displayed on at least one of the public virtual image acquisition devices according to each of the screen display sizes.

[0007] In the above scheme, if the perspective mode is single-player mode, determining the target position of the virtual image acquisition device to be moved includes: The system obtains the target speaker's first height, first horizontal body dimension, and first position in the virtual meeting; it also obtains the target non-speaking object's second position in the virtual meeting among all participants; and it obtains the vertical and horizontal viewing angles of the virtual image acquisition device. The target position is determined based on the first height, the first horizontal body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle.

[0008] In the above scheme, determining the target position based on the first height, the first horizontal body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle includes: Based on the first height and the vertical viewing angle, determine the first vertical distance between the target non-speaking object and the target speaking object among the non-speaking objects; Based on the first body lateral dimension and the horizontal viewing angle, determine the second horizontal distance between the target non-speaking object and the target speaking object; The larger of the first distance and the second distance is determined as the target distance; The target location is determined based on the first location, the relative position between the first location and the second location, and the target distance.

[0009] In the above scheme, if the view mode is a multi-user mode, determining the target position of the virtual image acquisition device to be moved includes: For each speaking group to be displayed in the speaking group, obtain the second height, second body horizontal dimension and third position of each speaking object in the speaking group to be displayed, and determine the target height based on multiple second heights, determine the target body horizontal dimension based on multiple second body horizontal dimensions, and determine the center position of all speaking objects in the speaking group to be displayed based on multiple third positions. Obtain the fourth position of the target non-speaking object in the virtual meeting, and obtain the vertical and horizontal viewing angles of the virtual image acquisition device; The target position is determined based on the target height, the target body lateral dimensions, the center position, the fourth position, the vertical viewing angle, and the horizontal viewing angle.

[0010] In the above scheme, the virtual conference control method further includes: When it is determined through skeletal point tracking technology that the speaker's lateral body dimensions have changed, the target position is redefined. When the number of speakers in any of the aforementioned speaking groups changes, the screen display size of each speaking group is redefined.

[0011] This application provides a virtual conference control device, the device comprising: The acquisition unit is used to determine the microphone status and voice signal of each participant in the virtual meeting, and to determine the speaker in the virtual meeting based on the microphone status and the voice signal. The first determining unit is configured to determine, based on the number of speaking participants, the voice output duration of each speaking participant, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant among the participants, to determine the speaker to be displayed on each private virtual image acquisition device and / or at least one of the public virtual image acquisition devices; wherein, the perspective mode includes a single-person mode and a multi-person mode. The second determining unit is used to determine the target position of the virtual image acquisition device to be moved; wherein the target position is a position that fully displays the upper body or full body of the speaker to be displayed; the virtual conference is also configured with at least one public virtual image acquisition device; the virtual image acquisition device to be moved includes: a private virtual image acquisition device corresponding to the non-speaking person among all participants and / or a public virtual image acquisition device used to display the secondary speaking group in the multi-person mode; The moving unit is used to control the virtual image acquisition device to be moved to move from its original position along a straight line to the target position, and to capture and display the image of the speaker to be shown at the moved position.

[0012] This application embodiment provides a virtual conference control device, the device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0013] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0014] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the method provided in this application.

[0015] The embodiments of this application have the following beneficial effects: By detecting the microphone status and voice output duration of the participants, the speaker is automatically identified, and the speaker to be displayed is intelligently determined in combination with the user's preset single-person mode or multi-person mode. The perspective switching can be triggered without manual operation by the user, which significantly improves the ease of use and response speed of the virtual meeting; and for the virtual image acquisition device to be moved, the target position that can be fully displayed to show the upper body or whole body of the speaker is calculated, and the device is controlled to move quickly along a straight line to the position for shooting and display. Thus, in any scenario (including multiple people speaking at the same time, a large number of AI users, or a large space), the speaker can be fully and clearly presented in the picture, which effectively overcomes the problem of related technologies that only adjust the direction of the line of sight and cannot guarantee the integrity of the framing; at the same time, through the adaptive switching between single-person mode and multi-person mode, it can support focusing on a single main speaker, and also support clearly displaying multiple speakers at the same time through clustering and screen size allocation, which solves the problem that the existing technology can only focus on a single person and cannot observe multiple people at the same time. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a virtual human in the virtual meeting control method provided in the embodiments of this application; Figure 2 This is another virtual human diagram in the virtual meeting control method provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the virtual meeting control method provided in this application embodiment; Figure 4 This is a schematic diagram of the conference screen display in the virtual conference control method provided in the embodiments of this application; Figure 5 This is a second flowchart illustrating the virtual meeting control method provided in this application embodiment; Figure 6 This is a third flowchart illustrating the virtual meeting control method provided in this application embodiment; Figure 7 This is a schematic diagram illustrating the display of a target speaker in a virtual conference control method provided in this application embodiment; Figure 8 This is a schematic diagram illustrating another method for displaying a target speaker in the virtual conference control method provided in this application embodiment; Figure 9 This is a schematic diagram of the display of speaking groups in the virtual conference control method provided in this application embodiment; Figure 10 This is a schematic diagram of the structure of the virtual conference control device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the virtual conference control device provided in the embodiments of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that in existing virtual meetings, after participants enter the meeting room, they use buttons to move a virtual persona to a designated location. Unless the user performs any further action, the virtual persona will maintain the same viewpoint. Figure 1 As shown; when someone speaks, if the perspective is inappropriate (refer to...) Figure 2 As shown, the virtual avatar faces different directions, and participants can only see the speaker's name and image displayed on the screen. If participants want to see the speaker's expressions or actions, they must manually switch the virtual avatar to a suitable position using buttons. This operation is very complex and time-consuming, and even after multiple attempts, it may still be impossible to find a suitable viewpoint, impacting the participant experience. This is especially true for presenters who are projecting their presentations; once the presenter has projected, if someone asks a question, it's impossible to immediately switch the virtual avatar's perspective to focus on the speaker. The way the presenter and speaker interact remains similar to traditional online video conferencing, relatively simplistic, and fails to highlight the advantages of virtual conferencing in simulating reality.

[0020] To address the aforementioned issues, the specific implementation of the relevant technology is as follows: During a virtual meeting, when a second user is detected speaking, the coordinates of the second user's corresponding second AI person in the virtual meeting scenario, as well as the coordinates and current viewpoints of the first AI persons corresponding to each of the other first users, are used to calculate the viewpoint adjustment parameters for each first AI person. Based on these parameters, the current viewpoints of each first AI person are adjusted to switch their gaze towards the second AI person. The display screen of the corresponding first user is then adjusted to show the second AI person. However, this technology has the following technical problems: 1. It cannot guarantee that the first AI person can clearly see the second AI person after adjusting their gaze, especially when there are many AI persons in a large space. 2. The first AI person's gaze cannot be dynamically adjusted according to the second AI person's actions and posture. For example, when the second AI person changes from a standing to a sitting posture, it may not be possible to see the second AI person clearly. 3. The solution only focuses on a single person; when multiple people are speaking, other people cannot be observed.

[0021] Based on this, embodiments of this application provide a virtual meeting control method, referring to... Figure 3 As shown, this method, applied to virtual conference control devices, may specifically include the following steps: Step 101: Determine the microphone status and voice signal of each participant in the virtual meeting, and determine the speaker in the virtual meeting based on the microphone status and voice signal.

[0022] In this embodiment, a virtual meeting can have multiple participants, and the participants can specifically refer to virtual humans (digital humans); the microphone status refers to whether the microphone of each participant in the virtual meeting is turned on (including on and off), which is the basic physical signal for determining who might be speaking; the speaking object is the object currently or has been outputting voice from all participants with their microphones on; during the virtual meeting, the microphone status of all participants in the virtual meeting can be continuously monitored. When a participant turns on their microphone (i.e., turns on the microphone) and a voice signal is detected, the participant is determined to be the speaking object. In this way, the two parameters of microphone status and voice signal together ensure that only the participant who is actually speaking will trigger the subsequent perspective change, avoiding invalid switching caused by accidental microphone turning on or environmental noise.

[0023] Step 102: Based on the number of speakers, the duration of each speaker's voice output, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant, determine the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device.

[0024] The perspective modes include single-player mode and multiplayer mode.

[0025] In this embodiment, the virtual conference room provides each participant with a private virtual image capture device and pre-sets multiple public virtual image capture devices. When a participant enters the virtual conference, they can activate their personal virtual camera (i.e., their private virtual image capture device) and display the view of their current location. (Refer to...) Figure 4 As shown; Private virtual image capture devices can be understood as each participant's personal image capture device (such as a camera). In a virtual meeting, it determines what the participant sees on their own screen, simulating each person's own eyes and perspective in a real meeting; Public virtual image capture devices are used to capture images from the best angle in complex scenarios (such as multiple people speaking simultaneously), for multiple participants to view; Perspective modes include single-person mode and multi-person mode, and the perspective mode is set by the participants based on their personal preferences. In single-person mode, participants want their screen to focus only on the most important speaker, rather than trying to display all speakers simultaneously, simulating the habit of people tending to focus on the current main speaker in a real meeting; In multi-person mode, participants want to see all speakers (or groups of speakers) on their screen simultaneously, rather than focusing only on one main speaker, simulating the scenario in a real meeting where multiple speakers need to be paid attention to at the same time; Voice output duration is the cumulative length of time each speaker speaks continuously; The speaker to be displayed is the final target object to be displayed on the screen, which can be a specific speaker or a group of speakers composed of multiple neighboring speakers.

[0026] In this embodiment, the number of speakers in a virtual meeting (i.e., whether it's one speaker or multiple speakers) can be counted, and the audio output duration of each speaker can be recorded. Then, the view mode set by the non-speaking participants in each meeting can be obtained. After comprehensive judgment, if there is only one speaker, that speaker is the one to be displayed. If there are multiple speakers, and the view mode of a non-speaking participant is single-person mode, a primary speaker can be selected as the speaker to be displayed based on the audio output duration of the multiple speakers. If there are multiple speakers, and the view mode of the non-speaking participants is multi-person mode, all speakers can be grouped into speaking groups, and all speaking groups can be identified as the speakers to be displayed. These speakers to be displayed can then be assigned to private virtual image capture devices (focusing on the main speaker or main speaking group) and / or public virtual image capture devices (displaying secondary speaking groups). This not only handles simple scenarios of single-person speaking but also effectively addresses complex multi-person speaking situations. Furthermore, it provides differentiated screen strategies based on the different preferences of the participants (focusing on one person or viewing the whole picture), improving flexibility and user-friendliness.

[0027] Step 103: For the virtual image acquisition device to be moved, determine the target location of the virtual image acquisition device to be moved.

[0028] The target location is the position that fully displays the upper body or full body of the speaker to be displayed; the virtual meeting is also equipped with at least one public virtual image acquisition device; the virtual image acquisition devices to be moved include: the private virtual image acquisition devices corresponding to the non-speaking participants among all participants and / or the public virtual image acquisition devices used to display the next speaking group in multi-person mode.

[0029] In this embodiment, the virtual image acquisition device to be moved refers to those virtual image acquisition devices that need to change their current position to achieve the best viewing effect, mainly including two parts: 1) private virtual image acquisition devices corresponding to all non-speaking participants; 2) public virtual image acquisition devices used to display secondary speaking groups in multi-person mode. It should be noted that the private virtual image acquisition devices corresponding to the speaking object usually do not move in order to maintain the stability of their own perspective. The target position refers to an optimal three-dimensional spatial coordinate calculated so that the virtual image acquisition device can completely and clearly capture the speaker to be displayed. Complete display specifically means: if the speaker to be displayed is standing, the entire body of the speaker to be displayed (from head to toe) must be captured; if the speaker to be displayed is sitting, the upper body of the speaker to be displayed must be captured.

[0030] In this embodiment, for each virtual image acquisition device determined to be moved, a new and better shooting position (i.e., target position) can be planned from its default original position. The target position is determined by comprehensively considering factors such as the virtual camera's viewing angle, the speaker's height, body width, and current location. It should be noted that at the target position, regardless of whether the speaker is standing or sitting, and regardless of their body shape, the target position can ensure that the speaker is completely and without omission in the frame. For example, for a standing speaker, the target position will be moved further away to capture the whole body; for a sitting speaker, the target position will be moved closer to focus on the upper half. In this way, the core problem of the prior art that it is impossible to guarantee a complete and clear view of the speaker is solved. Furthermore, by accurately calculating the target position through the algorithm, it is ensured that the user can immediately obtain the best viewing image with each viewpoint switch, avoiding the embarrassment and inefficiency of not being able to switch to the appropriate viewpoint when manually adjusting.

[0031] Step 104: Control the virtual image acquisition device to be moved to move from the original position to the target position along a straight line, and capture the image of the speaker to be displayed at the new position.

[0032] In this embodiment, moving along a straight line refers to the shortest path for the camera to move from one location to another. After calculating the precise target location, the relevant virtual image acquisition device can be directly controlled and smoothly moved from its current original position to the calculated target location along a straight line. Once reached, the virtual image acquisition device begins to work at the target location, capturing the image of the speaker to be presented and displaying this image on the screen of the corresponding participant in real time. It should be noted that the entire movement process is automatic, fast, and continuous. Users do not need to perform any manual operation and can see the image automatically adjusted to the best viewing angle.

[0033] The virtual meeting control method provided in this application automatically identifies the speaker by detecting the microphone status and voice output duration of the participants, and intelligently determines the speaker to be displayed by combining the user's preset single-person mode or multi-person mode. The viewpoint switching can be triggered without manual operation by the user, which significantly improves the ease of use and response speed of the virtual meeting. For the virtual image acquisition device to be moved, the method calculates the target position that can fully display the upper body or whole body of the speaker, and controls the device to move quickly along a straight line to the position for shooting and display. Thus, in any scenario (including multiple people speaking at the same time, a large number of AI users, or a large space), it can ensure that the speaker is fully and clearly presented in the picture, which effectively overcomes the problem of related technologies that can only adjust the direction of the line of sight and cannot guarantee the integrity of the framing. At the same time, through the adaptive switching between single-person mode and multi-person mode, it can support focusing on a single main speaker, as well as supporting the clear display of multiple speakers at the same time through clustering and screen size allocation, solving the problem that the existing technology can only focus on a single person and cannot observe multiple people at the same time.

[0034] Based on the foregoing embodiments, this application provides yet another virtual conference control method, referring to... Figure 5 and Figure 6 As shown, this method, applied to virtual conference control devices, may specifically include the following steps: Step 201: Determine the microphone status and voice signal of each participant in the virtual meeting, and determine the speaker in the virtual meeting based on the microphone status and voice signal.

[0035] It should be noted that after step 201, steps 202-203 can be executed, or steps 204-207 can be executed.

[0036] Step 202: If there are multiple speakers and the perspective mode is single-person mode, determine the target speaker from the multiple speakers based on the voice output duration of each speaker.

[0037] Among them, the speakers to be presented include the target speakers.

[0038] In this embodiment of the application, "multiple speakers" refers to two or more participants in a virtual meeting simultaneously turning on their microphones and outputting voice; "target speaker" refers to a single speaker selected after comprehensive judgment in a scenario where multiple people are speaking at the same time, and who is currently the focus of attention. When multiple participants are detected speaking simultaneously, and the perspective mode of a currently non-speaking participant is set to single-person mode, the system does not attempt to display everyone. Instead, a filtering mechanism is activated. This mechanism acquires and compares the audio output duration of all participants, and includes participants whose audio output duration exceeds a target threshold (e.g., speaking continuously for more than 3 seconds) as candidates. From these candidates, the target participant is determined according to a priority rule (e.g., the first speaker is given priority). Alternatively, the participant with the longest audio output duration can be directly selected as the target participant. This addresses the problem of related technologies that only focus on a single person, failing to observe the partial understanding of others when multiple people are speaking. It also avoids frequent and meaningless screen jumps when multiple people want to speak, ensuring that the participants' attention remains focused on the most important and continuous speaker. This aligns with human social habits and improves meeting focus and communication efficiency.

[0039] Step 203: Determine the target speaker to be displayed on each private virtual image capture device.

[0040] In this embodiment, after selecting a unique target speaker from multiple speakers, the private virtual image capture devices of all non-speaking participants (i.e., audience members) will be used to display the image of the target speaker. (Refer to...) Figure 7 (The target person being addressed while standing) and Figure 8 As shown (the target speaker is seated); that is, regardless of the number of listeners, all their screens will automatically switch to the same optimal viewing angle to watch the selected target speaker. This ensures that in complex multi-person speaking environments, the attention of all listeners can be guided to the same core speaker in a consistent and deterministic manner. This not only simplifies the control logic (eliminating the need to calculate different targets for each listener individually) but also provides listeners with a clear and consistent viewing experience—everyone can simultaneously focus on the most important speaker, thereby promoting collective attention and effective communication in virtual meetings.

[0041] It should be noted that the virtual meeting screen for the target speaker will retain the original screen; and refer to... Figure 6 As shown, when the duration of no speaking exceeds the target threshold, the display screen of each private virtual image acquisition device returns to the screen seen by each participant at their current location.

[0042] Step 204: If there are multiple speakers and the perspective mode is multi-person mode, group all speakers to obtain at least one speaking group.

[0043] Among them, the speakers to be presented include at least one speaking group.

[0044] In this embodiment, when multiple participants are detected speaking simultaneously, and the perspective mode of a non-speaking participant is set to multi-person mode, a clustering algorithm (such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN)) can be used to automatically divide spatially adjacent speakers into several speaking groups. Specifically, based on the coordinates of each speaker in the virtual meeting room, speakers who are close to each other are grouped into one speaking group, while speakers who are far away and cannot be clearly captured by the same camera are grouped into different groups. Finally, at least one (usually multiple) speaking groups are obtained. This solves the problem in the prior art that the solution only focuses on a single person and that multiple people cannot be clearly observed simultaneously when they are scattered. This application avoids the drawback of excessively zooming out the camera to capture multiple speakers in the distance, which makes each person appear very small and their expressions and movements unclear. Moreover, the grouping strategy allows speakers in each speaking group to share an optimal shooting perspective, thereby achieving the effect of clearly and completely seeing all speakers.

[0045] Step 205: Determine the total voice output duration of each speaking group based on the voice output duration of each speaking object within each speaking group, and determine the speaking group with the longest total voice output duration as the main speaking group, and determine the remaining speaking groups other than the main speaking group as secondary speaking groups.

[0046] In this embodiment, the total voice output duration is a quantitative indicator of the overall speaking activity of a speaking group, obtained by summing the voice output durations of each speaker in the group. The main speaking group refers to the group with the longest total voice output duration among all speaking groups, representing the most active speakers in the current meeting who need to be given the most attention. The secondary speaking groups refer to all other speaking groups besides the main speaking group, whose overall activity is relatively low, but they still need to be displayed on the screen to maintain the overall picture of the meeting.

[0047] In this embodiment, after grouping, multiple speaking groups are obtained. For each speaking group, the total voice output duration of all members in the group up to the current time is accumulated to calculate the total voice output duration of each group. Then, all total voice output durations are compared, and the group with the longest total voice output duration is identified and marked as the main speaking group. All other groups are automatically marked as secondary speaking groups. In this way, by assigning different visual weights to different speaking groups, that is, by defining the speaking group with the longest total speaking duration as the main speaking group, the current discussion focus of the meeting can be dynamically identified, providing a basis for decision-making in allocating the best viewing position and the largest screen to that group. Step 206: Determine the screen display size for each speaking group based on the total voice output duration.

[0048] In this embodiment, the screen display size refers to the proportion of screen area occupied by each speaking group on the final composite display screen. The speaking group with a longer total voice output duration has a larger screen display size, and the speaking group with a shorter total voice output duration has a smaller screen display size. After calculating the total voice output duration of each speaking group, the sum of the total voice output durations of all speaking groups can be calculated. Then, the total voice output duration of each speaking group is divided by the sum of the total voice output durations of all speaking groups (i.e., the total voice output duration of that speaking group / the sum of the total voice output durations of all speaking groups) to obtain the duration proportion (i.e., weight) of each speaking group. This weight directly determines the proportion of screen size that the speaking group should occupy on the screen. For example, if the total audio output duration of the main speaking group is 60 seconds, and the total audio output duration of one secondary speaking group is 20 seconds, then the screen display size ratio of these three speaking groups is 3:1:1. In this way, screen resources are allocated on demand, that is, the more active the speaking group and the longer the total audio output duration, the larger the screen size they receive. This allows the attention of the participants to be naturally guided to the core discussion points of the meeting. At the same time, the secondary speaking groups still receive a certain screen proportion to ensure the integrity of the information. This dynamic and non-uniform screen layout is more efficient and in line with communication logic than the equal-screen or displaying only one person in related technologies.

[0049] Step 207: Determine that the main speaking group is displayed on each private virtual image acquisition device, and that each speaking group is displayed on at least one public virtual image acquisition device according to the display size of each screen.

[0050] In this embodiment, for all participants who are not speaking, their individual private virtual image capture devices will be used uniformly to display the main speaking group's screen. This means that the main screen of each non-speaking participant will focus on the most active speaking group. For all secondary speaking groups, one or more public virtual image capture devices will be used to capture them separately. Specifically, if the number of secondary speaking groups is n, n (or n-1, if the main speaking group has already occupied a private device) public virtual cameras will be used. Each public virtual image capture device will be responsible for one secondary speaking group, and the screens of these secondary speaking groups will be laid out and displayed strictly according to the screen display size ratio (e.g., picture-in-picture or split-screen format), see reference. Figure 9 As shown, by distinguishing the uses of private and public devices, the display complexity problem in multi-person mode is solved. This allows each participant's main screen (private device) to focus on displaying the most important main speakers, ensuring the core viewing experience. Public devices are responsible for supplementing the display of other secondary speakers, ensuring the comprehensiveness of meeting information. This avoids a single screen appearing cluttered and crowded due to trying to display too much content, and allows participants to easily grasp the dynamics of all speakers by scanning their eyes, greatly improving the sense of presence and information acquisition efficiency of virtual meetings.

[0051] It should be noted that after step 203, steps 208-209 can be executed, and after step 207, steps 210-212 can be executed.

[0052] Step 208: If the perspective mode is single-person mode, obtain the first height, first horizontal body dimension and first position of the target speaker in the virtual meeting, and obtain the second position of the target non-speaking person in the virtual meeting among all participants, and obtain the vertical and horizontal perspective angles of the virtual image acquisition device.

[0053] In this embodiment, the first height refers to the height of the target speaker. The specific calculation method depends on their posture. If standing, the first height refers to the total height of the target speaker; if sitting, the first height refers to the height of the target speaker's upper body (i.e., from the top of the head to the center of the upper body). The first height can be denoted as... The first body lateral dimension refers to the maximum width of the target speaker's body on both the left and right sides, and can be denoted as... The first position is the three-dimensional spatial coordinates of the target speaker in the virtual meeting scene. When the target speaker is seated, the first position is defined as the center of the upper body; when the target speaker is standing, the first position is defined as the center of the whole body. The first position can be denoted as... The target non-speaking participant is any participant who is not currently speaking; the second position is the target non-speaking participant's three-dimensional spatial coordinates within the virtual meeting scenario, and can be denoted as... The vertical viewing angle is the size of the virtual camera's vertical viewing angle, and can be denoted as... The horizontal view angle is the size of the virtual camera's horizontal field of view, and can be denoted as... In one feasible implementation, the vertical viewing angle is set to 100 degrees, which is the maximum human viewing angle; the horizontal viewing angle is set to 200 degrees, which is the maximum human viewing angle.

[0054] In this embodiment of the application, the target speaker determined in single-person mode is first locked, and its first height, first body horizontal dimension and its first position in the virtual scene are obtained. Then, a non-speaking participant whose screen needs to be adjusted is randomly selected (i.e. the target non-speaking object), and its current second position is obtained. Then, the preset vertical and horizontal viewing angles of the virtual camera are obtained.

[0055] Step 209: Determine the target position based on the first height, the first body lateral dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle.

[0056] In this embodiment, the target position is the final three-dimensional spatial coordinate where the virtual image acquisition device to be moved needs to be located, and the target position can be denoted as... It should be noted that shooting at the target location ensures that the speaker's entire body (standing) or upper body (sitting) is fully and clearly presented in the image. The system can calculate the first person's height and vertical viewing angle, as well as the first person's horizontal dimensions and horizontal viewing angle. Based on these calculations, the target location is then determined using the first and second positions.

[0057] It should be noted that step 209 can be achieved in the following way: Step 209A1: Based on the first height and vertical viewing angle, determine the first vertical distance between the target non-speaking object and the target speaking object among the non-speaking objects.

[0058] In this embodiment, the first distance may refer to the minimum horizontal distance between the virtual camera of the target non-speaking object and the target speaking object; in order to fully capture the height of the target speaker in the vertical direction, the virtual camera corresponding to the target non-speaking object must maintain a certain distance from the target non-speaking object (i.e., the first distance, which can be denoted as...). Furthermore, formulas can be used to determine the first height and vertical viewing angle. The calculation yields the first distance; it should be noted that the first distance ensures that the top of the target speaker's head and the bottom of their feet (or the top of their upper body and the center point) fall exactly within the edge of the camera's vertical field of view.

[0059] Step 209A2: Based on the first body horizontal dimension and horizontal viewing angle, determine the second horizontal distance between the target non-speaking object and the target speaking object.

[0060] In this embodiment, the second distance refers to the minimum distance in the Z direction between the virtual camera of the target non-speaking object and the target speaking object; in order to fully capture the body width of the target speaker in the horizontal direction, a minimum distance also needs to be calculated, namely the second distance, which can be denoted as... Furthermore, formulas can be used to determine the first body's lateral dimensions and horizontal viewing angle. The calculation yields the second distance; it should be noted that the second distance ensures that the speaker's widest left and right sides fall exactly within the edge of the camera's horizontal field of view.

[0061] Step 209A3: Determine the larger of the first distance and the second distance as the target distance.

[0062] In this embodiment, to ensure that the shooting requirements in both vertical and horizontal directions are met simultaneously, the virtual image acquisition device must be moved to a distance that can simultaneously cover both height and width. Therefore, the maximum value of the first distance and the second distance is taken as the final safe distance, i.e., the target distance (which can be denoted as...). ),Right now .

[0063] It should be noted that by taking This ensures that the target speaker can be captured completely and clearly under any circumstances (tall, broad-shouldered, standing or sitting), thus solving the problem that existing technologies cannot guarantee a complete and clear view.

[0064] Step 209A4: Determine the target position based on the first position, the relative position between the first position and the second position, and the target distance.

[0065] In this embodiment of the application, the relative position between the first position and the second position, the target distance, and the Z-axis coordinate of the first position can be used as the basis. Determine the Z-axis coordinates of the target position. ), and based on the X-axis and Y-axis coordinates of the first position ( Determine the X and Y coordinates of the target location. ),Right now Specifically, to minimize the movement distance and ensure the target speaker is centered, the camera's X and Y coordinates are directly aligned with the center position of the target speaker. , Furthermore, it is necessary to specify the Z-axis coordinate of the target speaker ( ); Based on this, add or subtract a safety distance; specifically, compare the positions of the target unspeaking objects. ) and the location of the target speaker ( ):if (That is, the unspeakable target is behind the target speaker), then (That is, the camera moves backward); if (That is, the target who has not yet spoken is in front of the target who is speaking), then (That is, the camera moves forward) to obtain the target position. .

[0066] Step 210: If the perspective mode is multi-person mode, for each speaking group to be displayed in the speaking group, obtain the second height, second body horizontal dimension and third position of each speaking object in the speaking group to be displayed, and determine the target height based on multiple second heights, determine the target body horizontal dimension based on multiple second body horizontal dimensions, and determine the center position of all speaking objects in the speaking group to be displayed based on multiple third positions.

[0067] In this embodiment, the speaking group to be displayed refers to the speaking group that needs to be displayed after being grouped in multi-person mode. It may be the main speaking group with the longest total speaking time, or it may be other secondary speaking groups; the second height is the height of each speaker in the speaking group to be displayed, which can be denoted as... The second horizontal dimension is the maximum left and right width of each speaker's body in the speaking group to be displayed. The third position is the three-dimensional spatial coordinates of each speaker in the group to be displayed within the virtual meeting scene, denoted as... The target height is the maximum height selected from all the tallest members (i.e., the second tallest) in the group, to ensure the virtual camera can capture the entire tallest member. The target body's horizontal dimension is the maximum value among all the widest speakers in the group, chosen to ensure the virtual camera can capture the entire speaker group. This value is the second maximum horizontal dimension. The center position is the geometric center point in space of all speakers within the group to be displayed, which can be represented by three-dimensional coordinates. To express.

[0068] In this embodiment of the application, for a certain speaking group, all speaking objects in the speaking group are traversed, and the second height, second body horizontal dimension and third position of each speaking object in the group are obtained one by one. Then, the second height of all speaking objects in the group is compared, and the maximum value is selected as the target height. The second body horizontal dimension of all members in the group is compared, and the maximum value is selected as the target body horizontal dimension. Finally, the third position (X, Y and Z axis coordinates) of all speaking objects in the group are arithmetically averaged to obtain the coordinates of the center point.

[0069] Step 211: Obtain the fourth position of the target non-speaking participant in the virtual meeting, and obtain the vertical and horizontal viewing angles of the virtual image acquisition device.

[0070] In this embodiment, the fourth position refers to the three-dimensional spatial coordinates of the target non-speaking object in the virtual meeting scene in multi-person mode, denoted as... First, select a non-speaking person for whom you need to adjust a private virtual camera or assign a public virtual camera, and obtain their current location. Then, obtain the preset parameters of the virtual camera, namely the vertical viewing angle ( ) and horizontal viewing angle ( It should be noted that all camera viewpoint parameters ( , The camera setup is fixed and globally consistent, ensuring that all participants see a view with a consistent framing logic, thus avoiding differences in experience due to different camera settings.

[0071] Step 212: Determine the target position based on the target's height, horizontal body dimensions, center position, fourth position, vertical viewing angle, and horizontal viewing angle.

[0072] In this embodiment, the calculation logic for the target position within the speaking group is completely consistent with the single-person mode (steps 209A1-209A4), only the input data is replaced with "single person" instead of "speaking group". Specifically, to minimize the movement distance, the virtual camera position must be aligned with the center of the speaking group to be displayed. , And for the Z-axis coordinate of the target position ( The specific method for determining the distance between the target non-speaking object and the speaking group to be presented is as follows: First, based on the target's height and vertical viewing angle, determine the vertical distance between the target non-speaking object and the speaking group to be presented. This can be done using a formula. The calculated distance ensures that the top of the tallest person in the group (their head and feet, or the top of their upper body and center) falls within their vertical field of vision. Furthermore, based on the target's horizontal body dimensions and horizontal viewing angle, the horizontal distance between the unspeaking individual and the group to be presented can be determined using a formula. The calculation shows that this distance ensures the left and right boundaries of the widest speaker in the group fall within the horizontal field of view, after which it can be calculated from... and Choose the larger value as the final safe distance. ),Right now Thus, the target location is obtained. And if the Z-axis coordinate of the fourth position of the target non-speaking object ( The Z-axis coordinate of the center position of the group to be presented is greater than ( (If the target, who has not yet spoken, is behind the group to be shown speaking,) then the camera moves backward. ;if Less than (If the target person who has not yet spoken is in front of the group to be presented speaking), then the camera moves forward. Thus, 1) it solves the problem of multiple people watching simultaneously: it enables the simultaneous and complete filming of a group of speakers using a single virtual camera (or a public virtual camera), overcoming the shortcomings of existing technologies that can only focus on a single person or cannot guarantee the integrity of the group. 2) it ensures dynamic adaptability: when members of the group move (causing changes in target height or width) or when members are added or removed (causing changes in the center position), this step recalculates the target position based on the updated parameters, thereby achieving the intelligent effects of "dynamically adjusting the optimal position" and "dynamically adjusting the grouping and correspondingly adjusting the image when speakers are added or removed from the original group." 3) it balances efficiency and integrity: by taking... Ensuring a complete view, the system ensures efficient switching by aligning the center point and selecting the shortest movement path. This allows participants to automatically obtain a complete and clear view containing all speakers in complex scenarios with multiple speakers, without any manual operation, greatly enhancing the immersiveness and ease of use of virtual meetings.

[0073] It should be noted that step 213 can be executed after steps 209 and 212.

[0074] Step 213: Control the virtual image acquisition device to be moved to move from the original position to the target position along a straight line, and capture the image of the speaker to be displayed at the new position.

[0075] It should be noted that the above embodiments may further include the following steps: Step 214: When it is determined through skeletal point tracking technology that the speaker's lateral body size has changed, the target position is redefined.

[0076] In this embodiment, the virtual meeting system embeds a skeletal point tracking system. Skeletal point tracking technology is a human posture detection capability embedded in the virtual meeting system. This technology can identify and track the positions of key points of the virtual human's limbs (such as head, shoulder, elbow, wrist, knee, ankle, etc.) in real time, and analyze the virtual human's limb movements, posture changes (such as standing, sitting, waving, turning, etc.) and range of motion (such as the maximum width of the arm extension, the minimum height of the body squatting, etc.). The speaker refers to the participant who is currently speaking. In single-person mode, the speaker is the target speaker. In multi-person mode, the speaker can refer to any one or more speakers in the speaking group.

[0077] In this embodiment, a skeletal point tracking system continuously and in real-time detects the limb movements and range of motion of the current speaker (the target speaker in single-person mode or any speaker in a speaking group in multi-person mode). When the skeletal point tracking system detects that the speaker's limb movements (e.g., changing from naturally hanging arms to raising both arms, or changing from standing to sitting, or tilting the body to one side) cause a change in the maximum width of their body, a recalculation process is triggered. Specifically, for single-person mode, the current height of the target speaker is re-acquired. ) and body lateral dimensions ( ), and the current location of the target speaker ( For multi-person mode, the target height and target horizontal body dimensions of all speakers in the speaking group are recalculated, and the new center point coordinates of the entire speaking group are obtained. Then, substitute the newly acquired parameters into the geometric formula in step 209 (single-player mode) or step 212 (multiplayer mode) to recalculate the new target position. Then, the virtual image acquisition device to be moved can be controlled to move in a straight line from its current position to the newly calculated target position. This allows the system to adapt to the speaker's new posture after a change in body language. This achieves the following: 1) Ensures the integrity of the dynamic visuals: By using skeletal point tracking to perceive movement changes in real time and recalculate positions, it ensures that the speaker remains fully visible regardless of their actions (such as waving, standing, or sitting), achieving the effect of "the speaker can still be clearly and completely seen even in dynamic situations." This solves the problem in related technologies where virtual humans, once their initial perspective adjustment is complete, no longer adjust with changes in movement, leading to situations where the speaker's body parts might be cut off from the frame when making stretching or other movements. 2) Enhances immersion and naturalness: It simulates real-world meeting scenarios—when the speaker changes posture, the audience's gaze naturally follows. Automated dynamic perspective adjustment makes the virtual meeting experience more realistic and smooth, requiring no manual user intervention. 3) Supports complex scenarios: In multi-person mode, it can comprehensively consider the movement changes of all members in the group (e.g., one person raises their hand, another bends over), and recalculate a new optimal position that covers the entire range of movement, ensuring the integrity of the entire speaking group.

[0078] Step 215: When the number of speakers in any speaking group changes, redetermine the screen display size for each speaking group.

[0079] In this embodiment, the status (mic on, voice output) of all speakers in the virtual meeting and their spatial positions can be continuously monitored. When a new participant starts speaking (mic on and voice output), or an existing participant stops speaking (mic off or voice output duration below a threshold), causing a change in the number of participants in any speaking group (i.e., an increase or decrease in the number of participants), the DBSCAN clustering algorithm is re-run based on the three-dimensional coordinates of all current participants. This automatically groups nearby participants, meaning that existing speaking groups may be merged, split, or have members added or removed. For each newly formed speaking group, the speaking output duration of each member up to the current time is accumulated to obtain the group's output duration. The system calculates the new total audio output duration, then calculates the percentage of each group's total audio output duration based on all speaking groups. This percentage directly determines the screen size that group should occupy. The screen display is then rearranged and rendered according to the newly calculated screen size. The main speaking group (the group with the longest total audio output duration) continues to be displayed at a large size on the private virtual image capture device, while other secondary speaking groups adjust their display size proportionally on the public virtual image capture device. This achieves: 1) Adaptive grouping and screen layout: When the number of speakers increases or decreases, the system can automatically regroup and intelligently adjust the percentage of each group's screen, ensuring that all current speakers are clearly and orderly displayed. 2) Enhanced user experience: The entire process is fully automated. Participants can see the dynamically adjusted optimized screen containing all speakers in real time without any manual operation, greatly improving the usability and immersion of virtual meetings in multi-round dialogues and multi-person alternating speaking scenarios.

[0080] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0081] The virtual meeting control method provided in this application automatically identifies the speaker by detecting the microphone status and voice output duration of the participants, and intelligently determines the speaker to be displayed by combining the user's preset single-person mode or multi-person mode. The viewpoint switching can be triggered without manual operation by the user, which significantly improves the ease of use and response speed of the virtual meeting. For the virtual image acquisition device to be moved, the method calculates the target position that can fully display the upper body or whole body of the speaker, and controls the device to move quickly along a straight line to the position for shooting and display. Thus, in any scenario (including multiple people speaking at the same time, a large number of AI users, or a large space), it can ensure that the speaker is fully and clearly presented in the picture, which effectively overcomes the problem of related technologies that can only adjust the direction of the line of sight and cannot guarantee the integrity of the framing. At the same time, through the adaptive switching between single-person mode and multi-person mode, it can support focusing on a single main speaker, as well as supporting the clear display of multiple speakers at the same time through clustering and screen size allocation, solving the problem that the existing technology can only focus on a single person and cannot observe multiple people at the same time.

[0082] Based on the foregoing embodiments, this application provides a virtual conference control device, which can be applied to... Figure 3 and Figure 5 In the virtual conference control method provided in the corresponding embodiment, refer to Figure 10 As shown, the virtual conference control device 3 may include: an acquisition unit 31, a first determination unit 32, a second determination unit 33, and a movement unit 34, wherein: The acquisition unit 31 is used to determine the microphone status and voice signal of each participant in the virtual meeting, and to determine the speaker in the virtual meeting based on the microphone status and voice signal. The first determining unit 32 is used to determine the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant; wherein the perspective mode includes single-person mode and multi-person mode. The second determining unit 33 is used to determine the target position of the virtual image acquisition device to be moved; wherein, the target position is the position that fully displays the upper body or full body of the speaker to be displayed; at least one public virtual image acquisition device is also configured in the virtual meeting; the virtual image acquisition device to be moved includes: the private virtual image acquisition device corresponding to the non-speaking person among all participants and / or the public virtual image acquisition device used to display the secondary speaking group in multi-person mode; The moving unit 34 is used to control the virtual image acquisition device to be moved to move along a straight line from its original position to the target position, and to capture and display the image of the speaker at the new position.

[0083] In other embodiments of this application, the first determining unit 32 is further configured to perform the following steps: If there are multiple speakers and the perspective mode is single-person mode, the target speaker is determined from the multiple speakers based on the voice output duration of each speaker; among them, the speaker to be displayed includes the target speaker. Determine which speaker to display on each private virtual image capture device.

[0084] In other embodiments of this application, the first determining unit 32 is further configured to perform the following steps: If there are multiple speakers and the perspective mode is multi-person mode, all speakers are grouped to obtain at least one speaking group; among them, the speakers to be displayed include at least one speaking group. The total speech output duration of each speaking group is determined based on the speech output duration of each speaking object within each speaking group. The speaking group with the longest total speech output duration is determined as the main speaking group, and the other speaking groups besides the main speaking group are determined as secondary speaking groups. The screen display size for each speaking group is determined based on the total duration of multiple voice outputs. The main speaking group is displayed on each private virtual image capture device, and each speaking group is displayed on at least one public virtual image capture device according to the display size of each screen.

[0085] In other embodiments of this application, the second determining unit 33 is further configured to perform the following steps: Obtain the target speaker's first height, first horizontal body dimension, and first position in the virtual meeting; obtain the target non-speaking speaker's second position in the virtual meeting among all participants; and obtain the vertical and horizontal viewing angles of the virtual image acquisition device. The target position is determined based on the first height, the first horizontal body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle.

[0086] In other embodiments of this application, the second determining unit 33 is further configured to perform the following steps: Based on the first height and vertical viewing angle, determine the first vertical distance between the target non-speaking object and the target speaking object among the non-speaking objects; Based on the first body's horizontal dimensions and the horizontal viewing angle, determine the second horizontal distance between the target non-speaking object and the target speaking object; Determine the larger of the first and second distances as the target distance; The target position is determined based on the first position, the relative position between the first and second positions, and the target distance.

[0087] In other embodiments of this application, the second determining unit 33 is further configured to perform the following steps: For each speaking group to be displayed, obtain the second height, second body horizontal dimension, and third position of each speaking object in the speaking group to be displayed, and determine the target height based on multiple second heights, determine the target body horizontal dimension based on multiple second body horizontal dimensions, and determine the center position of all speaking objects in the speaking group to be displayed based on multiple third positions. Obtain the fourth position of the target non-speaking participant in the virtual meeting, and obtain the vertical and horizontal viewing angles of the virtual image acquisition device; The target position is determined based on the target's height, horizontal body dimensions, center position, fourth position, vertical viewing angle, and horizontal viewing angle.

[0088] In other embodiments of this application, the second determining unit 33 is further configured to perform the following steps: When the speaker's lateral body dimensions change as determined by skeletal point tracking technology, the target position is redefined. When the number of speakers in any speaking group changes, the screen display size of each speaking group is redefined.

[0089] It should be noted that a detailed explanation of the steps performed by each unit can be found in [reference needed]. Figure 3 and Figure 5 The virtual meeting control method provided in the corresponding embodiment will not be described in detail here.

[0090] The virtual meeting control device provided in this application automatically identifies the speaker by detecting the microphone status and voice output duration of the participants. It intelligently determines the speaker to be displayed by combining the user-preset single-person or multi-person mode. The viewpoint switching can be triggered without manual operation by the user, which significantly improves the ease of use and response speed of the virtual meeting. For the virtual image acquisition device to be moved, the device calculates the target position that can completely display the upper body or whole body of the speaker and controls the device to move quickly along a straight line to the position for shooting and display. Thus, in any scenario (including multiple people speaking at the same time, a large number of AI users, or a large space), it can ensure that the speaker is completely and clearly presented in the picture, which effectively overcomes the problem of related technologies that can only adjust the direction of the line of sight and cannot guarantee the integrity of the framing. At the same time, through the adaptive switching between single-person mode and multi-person mode, it can support focusing on a single main speaker, as well as supporting the clear display of multiple speakers at the same time through clustering and screen size allocation, solving the problem that the existing technology can only focus on a single person and cannot observe multiple people at the same time.

[0091] Based on the foregoing embodiments, embodiments of this application provide a virtual conference control device, which can be applied to... Figure 3 and Figure 5 In the virtual conference control method provided in the corresponding embodiment, refer to Figure 11 As shown, the virtual conference control device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein: Communication bus 43 is used to realize the communication connection between processor 41 and memory 42; The processor 41 is used to execute the virtual conference control program in the memory 42 to perform the following steps: Determine the microphone status and voice signal of each participant in the virtual meeting, and determine the speaker in the virtual meeting based on the microphone status and voice signal; Based on the number of speakers, the duration of each speaker's voice output, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant, determine the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device; wherein, the perspective mode includes single-person mode and multi-person mode. For the virtual image acquisition device to be moved, determine the target position of the virtual image acquisition device to be moved; wherein, the target position is the position that fully displays the upper body or full body of the speaker to be displayed; at least one public virtual image acquisition device is also configured in the virtual meeting; the virtual image acquisition device to be moved includes: the private virtual image acquisition device corresponding to the non-speaking person among all participants and / or the public virtual image acquisition device used to display the secondary speaking group in multi-person mode; The virtual image acquisition device is controlled to move from its original position to the target position along a straight line, and then captures and displays the image of the speaker at the new position.

[0092] In other embodiments of this application, the processor 41 is configured to execute the virtual conference control program in the memory 42 to determine the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the view mode of the private virtual image acquisition device corresponding to each participant, in order to achieve the following steps: If there are multiple speakers and the perspective mode is single-person mode, the target speaker is determined from the multiple speakers based on the voice output duration of each speaker; among them, the speaker to be displayed includes the target speaker. Determine which speaker to display on each private virtual image capture device.

[0093] In other embodiments of this application, the processor 41 is configured to execute the virtual conference control program in the memory 42 to determine the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device based on the number of speakers, the voice output duration of each speaker, and the view mode of the private virtual image acquisition device corresponding to each participant, in order to achieve the following steps: If there are multiple speakers and the perspective mode is multi-person mode, all speakers are grouped to obtain at least one speaking group; among them, the speakers to be displayed include at least one speaking group. The total speech output duration of each speaking group is determined based on the speech output duration of each speaking object within each speaking group. The speaking group with the longest total speech output duration is determined as the main speaking group, and the other speaking groups besides the main speaking group are determined as secondary speaking groups. The screen display size for each speaking group is determined based on the total duration of multiple voice outputs. The main speaking group is displayed on each private virtual image capture device, and each speaking group is displayed on at least one public virtual image capture device according to the display size of each screen.

[0094] In other embodiments of this application, the processor 41 is used to execute the virtual conference control program in the memory 42 in a single-person view mode. For the virtual image acquisition device to be moved, the processor determines the target position of the virtual image acquisition device to be moved in order to implement the following steps: Obtain the target speaker's first height, first horizontal body dimension, and first position in the virtual meeting; obtain the target non-speaking speaker's second position in the virtual meeting among all participants; and obtain the vertical and horizontal viewing angles of the virtual image acquisition device. The target position is determined based on the first height, the first horizontal body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle.

[0095] In other embodiments of this application, the processor 41 is used to execute the virtual conference control program in the memory 42 to determine the target position based on a first height, a first lateral body dimension, a first position, a second position, a vertical viewing angle, and a horizontal viewing angle, in order to achieve the following steps: Based on the first height and vertical viewing angle, determine the first vertical distance between the target non-speaking object and the target speaking object among the non-speaking objects; Based on the first body's horizontal dimensions and the horizontal viewing angle, determine the second horizontal distance between the target non-speaking object and the target speaking object; Determine the larger of the first and second distances as the target distance; The target position is determined based on the first position, the relative position between the first and second positions, and the target distance.

[0096] In other embodiments of this application, the processor 41 is used to execute the virtual conference control program in the memory 42 in a multi-person view mode. For the virtual image acquisition device to be moved, the processor determines the target position of the virtual image acquisition device to be moved in order to implement the following steps: For each speaking group to be displayed, obtain the second height, second body horizontal dimension, and third position of each speaking object in the speaking group to be displayed, and determine the target height based on multiple second heights, determine the target body horizontal dimension based on multiple second body horizontal dimensions, and determine the center position of all speaking objects in the speaking group to be displayed based on multiple third positions. Obtain the fourth position of the target non-speaking participant in the virtual meeting, and obtain the vertical and horizontal viewing angles of the virtual image acquisition device; The target position is determined based on the target's height, horizontal body dimensions, center position, fourth position, vertical viewing angle, and horizontal viewing angle.

[0097] In other embodiments of this application, the processor 41 is used to execute a virtual conference control method of the virtual conference control program in the memory 42 to implement the following steps: When the speaker's lateral body dimensions change as determined by skeletal point tracking technology, the target position is redefined. When the number of speakers in any speaking group changes, the screen display size of each speaking group is redefined.

[0098] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 3 and Figure 5 The virtual meeting control method provided in the corresponding embodiment will not be described in detail here.

[0099] The virtual meeting control device provided in this application automatically identifies the speaker by detecting the microphone status and voice output duration of the participants. It intelligently determines the speaker to be displayed by combining the user-preset single-person or multi-person mode. The viewpoint switching can be triggered without manual operation by the user, which significantly improves the ease of use and response speed of the virtual meeting. For the virtual image acquisition device to be moved, the device calculates the target position that can fully display the upper body or whole body of the speaker and controls the device to move quickly along a straight line to the position for shooting and display. This ensures that the speaker is fully and clearly presented in the picture in any scenario (including when multiple people speak at the same time, when there are many people or when the space is large), effectively overcoming the problem that related technologies can only adjust the direction of the line of sight and cannot guarantee the integrity of the framing. At the same time, through the adaptive switching between single-person mode and multi-person mode, it can support focusing on a single main speaker or clearly displaying multiple speakers at the same time through clustering and screen size allocation, solving the problem that the existing technology can only focus on a single person and cannot observe multiple people at the same time.

[0100] Based on the foregoing embodiments, this application provides a computer program product, including a computer program, which, when executed by a processor, implements... Figure 3 and Figure 5 The steps of the virtual conference control method provided in the corresponding embodiment.

[0101] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to achieve... Figure 3 and Figure 5 The steps of the virtual conference control method provided in the corresponding embodiment.

[0102] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A virtual meeting control method, characterized in that, The method includes: Determine the microphone status and voice signal of each participant in the virtual meeting, and determine the speaker in the virtual meeting based on the microphone status and the voice signal; Based on the number of speakers, the voice output duration of each speaker, and the perspective mode of the private virtual image acquisition device corresponding to each non-speaking participant, the speaker to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device is determined; wherein, the perspective mode includes single-person mode and multi-person mode; For the virtual image acquisition device to be moved, determine the target position of the virtual image acquisition device to be moved; wherein, the target position is the position that fully displays the upper body or full body of the speaker to be displayed; the virtual meeting is also configured with at least one public virtual image acquisition device; the virtual image acquisition device to be moved includes: the private virtual image acquisition device corresponding to the non-speaking object among all participants and / or the public virtual image acquisition device used to display the secondary speaking group in the multi-person mode; The virtual image acquisition device is controlled to move from its original position to the target position along a straight line, and then captures and displays an image of the speaker at the new position.

2. The method according to claim 1, characterized in that, The process of determining the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device, based on the number of speakers, the duration of each speaker's voice output, and the viewpoint mode of the private virtual image acquisition device corresponding to each participant, includes: If there are multiple speakers and the perspective mode is single-person mode, a target speaker is determined from the multiple speakers based on the voice output duration of each speaker; wherein, the speaker to be displayed includes the target speaker. The target speaker is determined to be displayed on each of the aforementioned private virtual image acquisition devices.

3. The method according to claim 1, characterized in that, The process of determining the speakers to be displayed on each private virtual image acquisition device and / or at least one public virtual image acquisition device, based on the number of speakers, the duration of each speaker's voice output, and the viewpoint mode of the private virtual image acquisition device corresponding to each participant, includes: If the number of speakers is multiple and the perspective mode is a multi-person mode, all speakers are grouped to obtain at least one speaking group; wherein, the speakers to be displayed include the at least one speaking group; The total voice output duration of each speaking group is determined based on the voice output duration of each speaking object within each speaking group. The speaking group with the longest total voice output duration is determined as the main speaking group, and the other speaking groups besides the main speaking group are determined as secondary speaking groups. The screen display size for each speaking group is determined based on the total duration of multiple voice outputs. The main speaking group is displayed on each of the private virtual image acquisition devices, and each of the secondary speaking groups is displayed on at least one of the public virtual image acquisition devices according to each of the screen display sizes.

4. The method according to claim 2, characterized in that, If the view mode is single-player mode, determining the target position of the virtual image acquisition device to be moved includes: The system obtains the target speaker's first height, first horizontal body dimension, and first position in the virtual meeting; it also obtains the target non-speaking object's second position in the virtual meeting among all participants; and it obtains the vertical and horizontal viewing angles of the virtual image acquisition device. The target position is determined based on the first height, the first horizontal body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle.

5. The method according to claim 4, characterized in that, Determining the target position based on the first height, the first lateral body dimension, the first position, the second position, the vertical viewing angle, and the horizontal viewing angle includes: Based on the first height and the vertical viewing angle, determine the first vertical distance between the target non-speaking object and the target speaking object among the non-speaking objects; Based on the first body lateral dimension and the horizontal viewing angle, determine the second horizontal distance between the target non-speaking object and the target speaking object; The larger of the first distance and the second distance is determined as the target distance; The target location is determined based on the first location, the relative position between the first location and the second location, and the target distance.

6. The method according to claim 3, characterized in that, If the view mode is a multi-user mode, determining the target position of the virtual image acquisition device to be moved includes: For each speaking group to be displayed in the speaking group, obtain the second height, second body horizontal dimension and third position of each speaking object in the speaking group to be displayed, and determine the target height based on multiple second heights, determine the target body horizontal dimension based on multiple second body horizontal dimensions, and determine the center position of all speaking objects in the speaking group to be displayed based on multiple third positions. Obtain the fourth position of the target non-speaking object among the participants in the virtual meeting, and obtain the vertical and horizontal viewing angles of the virtual image acquisition device; The target position is determined based on the target height, the target body lateral dimensions, the center position, the fourth position, the vertical viewing angle, and the horizontal viewing angle.

7. The method according to claim 1, characterized in that, The method further includes: When it is determined through skeletal point tracking technology that the speaker's lateral body dimensions have changed, the target position is redefined. When the number of speakers in any of the aforementioned speaking groups changes, the screen display size of each speaking group is redefined.

8. A virtual conference control device, characterized in that, The device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that can be executed by one or more processors to implement the steps of the virtual conference control method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the virtual conference control method according to any one of claims 1 to 7.

Images