Synchronous display method and device of video conference and storage medium

Abstract

The invention discloses a synchronous display method and device of a video conference and a storage medium, relates to the technical field of video conferences, and discloses the synchronous display method of the video conference, comprising the following steps: determining position information of each participant in a panoramic image based on the panoramic image collected by a panoramic lens; determining a close-up shot set in which each participant falls based on the position information and a spatial mapping relationship between the panoramic shot and each close-up shot; determining a target close-up shot from the close-up shot set according to the imaging quality of the participants in the close-up shots for the participants falling into the plurality of close-up shots; and generating a composite video picture containing all the participants based on close-up pictures of the participants output by the target close-up shot. According to the method and the device, the panoramic image is taken as a reference, and spatial mapping and imaging quality optimization are combined, so that repeated display of participants in a multi-lens cross area is avoided, and clear and non-redundant synchronous picture display is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video conferencing, and particularly relates to a synchronous display method of video conference, a device and a storage medium. BACKGROUND

[0002] In a video conference scene, in order to record all participants completely, a single or multiple cameras are usually used for shooting. However, when a participant is in an intersection shooting area of multiple cameras, the image of the participant will appear repeatedly in the final picture, so that the video conference system is difficult to stably track and synchronously present all participants in parallel, and the accuracy and overall visual effect of the conference picture are affected.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a synchronous display method of video conference, a device and a storage medium, which aims to solve the technical problem that the same participant is repeatedly displayed in a composite picture due to multiple camera intersection shooting.

[0005] To achieve the above purpose, the present application provides a synchronous display method of video conference, which is applied to a video conference system, the video conference system includes one panorama lens and multiple close-up lenses, and the synchronous display method of video conference includes: determining position information of each participant in a panorama image collected by the panorama lens based on the panorama image; determining a close-up lens set in which each participant falls based on the position information and a spatial mapping relationship between the panorama lens and each close-up lens; for the participant falling into multiple close-up lenses, determining a target close-up lens from the close-up lens set according to imaging quality of the participant in each close-up lens; generating a composite video picture containing all participants based on a close-up picture of the participant output by the target close-up lens.

[0006] In an embodiment, the step of determining the close-up lens set in which each participant falls based on the position information and the spatial mapping relationship between the panorama lens and each close-up lens includes: converting the position information of the participant in the panorama image to an image coordinate system of each close-up lens through the spatial mapping relationship to obtain mapping coordinates of the participant in each close-up lens; Wherein, the location information is the coordinates of the center point of the face detection box corresponding to the participant in the panoramic image, and the mapped coordinates are the coordinates of the center point after mapping in the image coordinate system of each close-up shot; Based on the mapped coordinates, determine whether the attendee is within the field of view of each of the close-up shots; The set of close-up shots is determined based on the determination results of whether the attendees are within the field of view of each close-up shot.

[0007] In one embodiment, before the step of determining the set of close-up shots in which each participant falls based on the location information and the spatial mapping relationship between the panoramic lens and each of the close-up shots, the synchronous display method of the video conference further includes: The panoramic lens and each of the close-up lenses respectively acquire calibration images containing multiple calibration reference points; Based on the corresponding image coordinates of each calibration reference point in the calibration image in the panoramic lens and each of the close-up shots, calculate the homography transformation matrix between the panoramic lens and each of the close-up shots; Based on the homography transformation matrix, the aspect ratio coefficient of each close-up shot relative to the panoramic shot is determined; Based on the homography transformation matrix and the image scaling factor, a spatial mapping relationship is established between the panoramic lens and each of the close-up shots.

[0008] In one embodiment, the step of determining a target close-up shot from the set of close-up shots based on the image quality of the attendee in each of the multiple close-up shots includes: Obtain the currently output video frame of each of the close-up shots in the set of close-up shots; Based on the video frames, calculate the image quality score for each close-up shot of the attendee; The close-up shot with the highest image quality score is identified as the target close-up shot.

[0009] In one embodiment, before the step of generating a composite video image containing all participants based on the close-up shots of the participants output from the target close-up shot, the synchronous display method of the video conference further includes: Based on the facial recognition results of the close-up images, a corresponding identity identifier is assigned to each of the attendees; When acquiring the close-up image output from the target close-up shot, a mapping relationship is established between the identity identifier and the corresponding close-up image; Based on the identity identifier and the mapping relationship, the close-up images associated with the same identity identifier are deduplicated to avoid the same participant being displayed repeatedly in the composite video image.

[0010] In one embodiment, the step of generating a composite video image containing all participants based on the close-up shots of the participants output from the target close-up shot includes: Obtain the actual coordinates of the participants in the close-up image output by the target close-up shot; Based on the actual coordinates and the spatial mapping relationship, determine the global mapping coordinates of the participants in the panoramic image; Obtain the center coordinates of the face detection bounding boxes corresponding to each of the attendees in the panoramic image; Calculate the spatial distance between the global mapping coordinates and each of the center coordinates, and determine the face detection box with the smallest spatial distance as the target detection box corresponding to the close-up image; The composite video frame is generated based on the correspondence between the target detection box and the close-up image.

[0011] In one embodiment, the step of generating the composite video frame based on the correspondence between the target detection box and the close-up shot includes: For each face detection box in the panoramic image, if there is no close-up image associated with the face detection box, then based on the position of the face detection box in the panoramic image, the corresponding display image is cropped from the panoramic image. The composite video frame is generated based on the correspondence between the target detection box, the close-up shot, and the display shot.

[0012] In one embodiment, after the step of generating the composite video image based on the correspondence between the target detection box and the close-up image, the synchronous display method for the video conference further includes: When the participant is detected to have moved out of the field of view of the current target close-up, the new target close-up that the participant will enter is predicted based on the participant's movement trajectory in the previous tracking cycle. The close-up video stream corresponding to the attendee is switched from the current target close-up to the new target close-up, and the composite video image is generated based on the close-up image output by the new target close-up after the switch.

[0013] This application embodiment also provides a synchronous display device for video conferencing, the synchronous display device for video conferencing includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the synchronous display method for video conferencing as described above.

[0014] This application embodiment also provides a storage medium, which is a computer-readable storage medium, and stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of the synchronous display method for video conferencing as described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application uses panoramic images to uniformly perceive the location information of all participants and combines it with the pre-calibrated spatial mapping relationship between panoramic lenses and various close-up shots to accurately determine the set of close-up shots that each participant falls into, thereby establishing a reliable correlation between the global and local perspectives. By introducing an imaging quality evaluation mechanism for the same participant who falls into multiple close-up shots, the target close-up shot is selected from the corresponding set of close-up shots, effectively avoiding multiple duplicate outputs caused by overlapping fields of view. Furthermore, by generating composite video images using only the close-up shots output by the determined target close-up shots, it ensures that each participant has one and only one clear, high-quality display window in the final image. In addition, this application uses panoramic images as a unified scheduling benchmark to achieve the coordinated allocation of multi-lens resources and the unique binding of image sources, fundamentally solving the problems of repeated participant images and unstable tracking in multi-camera cross-shooting scenarios, and achieving a synchronous parallel display effect for all participants. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the synchronous display method for video conferencing provided in Embodiment 1 of this application; Figure 2 A schematic diagram of the homography transformation calibration coordinate process provided for Embodiment 1 of the video conferencing synchronization display method of this application; Figure 3 A flowchart illustrating the second embodiment of the video conferencing synchronization display method of this application; Figure 4 A flowchart illustrating the third embodiment of the video conferencing synchronization display method of this application; Figure 5 A simplified flowchart illustrating the synchronous display method for video conferencing in Embodiment 3 of this application; Figure 6 This is a schematic diagram of the hardware operating environment of the video conferencing synchronous display device involved in the synchronous display method of video conferencing in the embodiments of this application.

[0017] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0019] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0020] In video conferencing scenarios, single or multiple cameras are typically used to capture all participants completely. However, when participants are in the overlapping shooting areas of multiple cameras, their images may appear repeatedly in the final frame. This makes it difficult for the video conferencing system to stably track and synchronize all participants in parallel, affecting the accuracy and overall viewing experience of the meeting footage.

[0021] In view of the above problems, this application proposes a synchronous display method for video conferencing. The method involves determining the position information of each participant in the panoramic image captured by the panoramic lens; determining the set of close-up shots in which each participant falls based on the position information and the spatial mapping relationship between the panoramic lens and each close-up shot; for participants falling in multiple close-up shots, determining a target close-up shot from the set of close-up shots based on the imaging quality of each participant in each close-up shot; and generating a composite video image containing all participants based on the close-up shots of the participants output from the target close-up shots.

[0022] This application provides a solution that, by uniformly perceiving the location information of all participants based on panoramic images and combining the pre-calibrated spatial mapping relationship between panoramic lenses and various close-up shots, accurately determines the set of close-up shots in which each participant falls, thereby establishing a reliable correlation between the global and local perspectives. For the same participant falling into multiple close-up shots, an imaging quality evaluation mechanism is introduced to select the target close-up shot from its corresponding set of close-up shots, effectively avoiding multiple duplicate outputs caused by overlapping fields of view. Furthermore, by generating composite video images using only the close-up shots output from the determined target close-up shots, it ensures that each participant has one and only one clear, high-quality display window in the final image. In addition, this application uses panoramic images as a unified scheduling benchmark to achieve collaborative allocation of multi-lens resources and unique binding of image sources, fundamentally solving the problems of image duplication and unstable tracking of participants in multi-camera cross-shooting scenarios, and achieving a synchronous parallel display effect for all participants.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a computer, video conferencing terminal, cloud processing platform, etc., or an electronic device, video conferencing synchronous display device, video conferencing system, etc., capable of realizing the above functions. The following description uses a video conferencing system as an example to illustrate this embodiment and the subsequent embodiments.

[0024] For the synchronous display method of video conferencing proposed in the first embodiment of this application, please refer to... Figure 1 The method includes steps S10 to S40: Step S10: Based on the panoramic image captured by the panoramic lens, determine the position information of each participant in the panoramic image.

[0025] It should be noted that a panoramic lens refers to a fixed camera device that covers the entire conference room scene with a wide field of view, used to acquire a panoramic image containing all participants; a close-up lens refers to a lens used to capture high-definition images of specific participants, with a smaller field of view, usually focusing on one or a few participants, and can provide higher resolution images; and location information refers to the coordinate data of the participants in the panoramic image coordinate system.

[0026] It is worth noting that coordinate data can also be represented based on a pixel coordinate system, and the specific coordinate system used depends on the actual application requirements and system design. For example, the top left corner of the panoramic image can be used as the origin, and the center point coordinates (x, y) of the face detection box corresponding to the participant can be used as their position information.

[0027] As a feasible implementation method, a deep learning-based object detection model (such as YOLO or Faster R-CNN) is used to perform real-time face detection on the panoramic images captured by the panoramic camera, and outputs the face detection box corresponding to each participant. The face detection box is defined by the coordinates of its upper left corner (x, y) and lower right corner (x', y'). Then, the geometric center coordinates (x0, y0) of the face detection box are calculated, where x0=(x+x') / 2 and y0=(y+y') / 2, and the coordinates (x0, y0) are used as the position information of the participant in the panoramic image coordinate system.

[0028] Understandably, given the potentially dense distribution and overlapping of attendees in a meeting setting, relying solely on close-up shots could easily lead to missed attendees or interrupted identity tracking. However, acquiring the global location information of all attendees through a panoramic camera effectively avoids tracking loss caused by blind spots in localized perception, thereby improving the completeness and accuracy of subsequent target association and camera allocation.

[0029] Step S20: Based on the location information and the spatial mapping relationship between the panoramic lens and each of the close-up shots, determine the set of close-up shots in which each participant falls.

[0030] It should be noted that the spatial mapping relationship refers to the geometric correspondence between the panoramic image coordinate system and the coordinate systems of each close-up image, which can be modeled using a homography transformation matrix. After acquiring the global position information of all participants through the panoramic lens, this spatial mapping relationship can be used to project the position coordinates of the participants in the panoramic image onto the image plane of each close-up shot, obtaining their mapped coordinates in each close-up shot, i.e., their theoretical imaging positions. Based on these mapped coordinates, the system can determine whether the participants are within the field of view of certain close-up shots, thus providing a reliable basis for lens selection for subsequent target tracking and high-quality close-up shot acquisition.

[0031] Additionally, a close-up shot set refers to a collection of one or more close-up shots that capture a complete or partial image of the attendee at the current moment. If the attendee is covered by only one close-up shot, the set contains only one close-up shot; if covered by multiple close-up shots, the set contains multiple close-up shots.

[0032] Understandably, since multiple close-up shots may have overlapping fields of view, the same attendee may be captured by multiple close-up shots simultaneously. Therefore, accurately determining the set of close-up shots to which an attendee belongs through spatial mapping can provide a candidate basis for subsequent selection of the target close-up shot. Based on this, the imaging effect of different close-up shots on the same attendee can be quantitatively evaluated, taking into account factors such as distance from the image boundary, degree of occlusion, and compositional integrity. From this, the close-up shot with the best imaging quality and avoiding edge cropping can be selected as the final target close-up shot, thereby ensuring the visual presentation effect of the output image.

[0033] As one possible implementation, steps S201 to S204 are included before step S20: Step S201: Acquire calibration images containing multiple calibration reference points using the panoramic lens and each of the close-up lenses.

[0034] It should be noted that a calibration reference point refers to a visual marker with a known location in physical space and that is easily detectable in an image. In this embodiment, multiple calibration reference points with fixed spatial locations are pre-arranged in the conference room. These calibration reference points are evenly distributed in key areas of the conference room, especially the field of view coverage areas and overlapping areas of each close-up shot. Markers with bright colors and clear outlines are selected as calibration reference points to ensure that they can be accurately identified in images captured by both panoramic and close-up shots.

[0035] To enhance calibration accuracy, calibration images can be acquired multiple times under different poses (such as translation and rotation) to obtain multiple sets of common viewpoint data, providing sufficient constraints for subsequent parameter solving.

[0036] Step S202: Calculate the homography transformation matrix between the panoramic lens and each of the close-up shots based on the corresponding image coordinates of each calibration reference point in the calibration image in the panoramic lens and each of the close-up shots.

[0037] It should be noted that the homography transformation matrix is ​​a 3×3 linear transformation matrix used to describe the projective geometric relationship between image planes from different viewpoints under the assumption of coplanarity. It can map a point on one image plane to the corresponding position on another image plane.

[0038] In this embodiment, corner detection algorithms (such as Harris corner and Shi-Tomasi) can be used to accurately locate the coordinates of each calibration reference point in the panoramic and close-up images. Then, the homography transformation matrix can be solved by combining the Direct Linear Transformation (DLT) algorithm with least squares optimization.

[0039] Specifically, for each pair of shots (e.g., a panoramic shot and close-up shot 1), at least three pairs of non-collinear corresponding points are selected, a homogeneous system of equations is constructed, and the initial homography matrix is ​​solved. To further improve accuracy, the RANSAC (RANdom SAmpleConsensus) algorithm is used to remove incorrectly matched point pairs. This process needs to be performed separately for all close-up shots, ultimately obtaining a complete set of homography transformation matrices for subsequent cross-shot coordinate transformations.

[0040] Step S203: Based on the homography transformation matrix, determine the aspect ratio coefficient of each close-up shot relative to the panoramic lens.

[0041] It should be noted that the aspect ratio factor refers to the scaling factor of a unit distance in a close-up image relative to the distance in the panoramic image in both horizontal and vertical directions. It is used to quantify the differences in imaging scale between different lenses. This factor is not a global constant but relies on the linear approximation characteristics of homography in local regions. Therefore, when calculating the aspect ratio factor, it can be calculated separately for different local regions in the panoramic image, thus more accurately reflecting the actual proportional relationship of each local region and improving mapping accuracy. In this way, scale distortion caused by lens characteristics can be corrected more precisely, allowing the position coordinates in the close-up image to be mapped back to their corresponding positions in the panoramic image more accurately.

[0042] For example, to facilitate understanding of the calculation method of the screen ratio coefficient in this embodiment, please refer to... Figure 2 The diagram shown illustrates the coordinate calibration process for homography transformation. Specifically: This embodiment uses homography to map the preset specific coordinate points in each close-up shot to the panoramic image coordinate system, thereby obtaining the calibration point coordinates of each close-up shot in the panoramic image, and calculating the screen ratio coefficients of the coordinate points of different areas in each close-up shot image relative to the panoramic image in the horizontal and vertical directions.

[0043] Taking close-up shot 1 as an example, its resolution is 3840×2160. There are three calibration reference points in the close-up shot: the upper left corner has coordinates (960, 540), the center point has coordinates (1920, 1080), and the upper right corner has coordinates (2880, 540). After homography transformation, the corresponding coordinates of these three points in the panoramic image are (x1, y1), (x2, y2), and (x3, y3), respectively. Therefore, the horizontal aspect ratio coefficient β on the left side based on the center point (x2, y2) can be calculated. left =(1920-960) / (x2-x1), the horizontal aspect ratio β on the right side of the image. right =(2880-1920) / (x3-x2); Similarly, the vertical aspect ratio coefficient γ = (1080-540) / (y2-y1). Due to perspective distortion, the aspect ratios on the left and right sides may differ slightly. Therefore, this embodiment uses a region-based calculation method to effectively improve local mapping accuracy. In addition, for other close-up shots (such as close-up shot 2 and close-up shot 3), the aspect ratio coefficients of their respective regions can be calculated separately using the same method for subsequent spatial mapping.

[0044] Step S204: Based on the homography transformation matrix and the image scaling factor, establish the spatial mapping relationship between the panoramic lens and each of the close-up shots.

[0045] In this embodiment, a mapping model from the close-up image coordinate system to the panoramic image coordinate system is constructed by combining the homography transformation matrix and the image scale factor. Specifically, for the coordinates of any point P in the close-up image, it is first projected to the initial coordinates P' of the panoramic image using the homography transformation matrix. Then, P' is fine-tuned according to the image scale factor corresponding to the area where the point is located to compensate for scale errors caused by perspective distortion. For example, in the left area of ​​close-up 1, its corresponding β is used. leftγ is used to scale and correct the horizontal and vertical directions, making the mapped coordinates closer to their actual physical locations. This spatial mapping can be used in subsequent steps to reverse-map the position coordinates of participants in the panoramic image to the coordinate system of the close-up shot, thereby determining whether they are within the field of view of a particular close-up shot. For example... Figure 2 As shown, each of the three close-up shots has a corresponding coverage area in the panoramic image. Through the above mapping relationship, spatial alignment and target association across shots can be achieved, thus providing reliable technical support for subsequent multi-shot collaborative tracking and composite video image generation.

[0046] This implementation method completes precise geometric calibration between multiple lenses based on visual information from multiple calibration reference points, establishing a stable and reliable mapping relationship between panoramic images and close-up shots. This not only improves the accuracy of target positioning but also enhances the system's adaptability and robustness in complex conference scenarios.

[0047] As another feasible implementation, a calibration method based on camera intrinsic and extrinsic parameters can be used to determine the spatial mapping relationship between the panoramic lens and each close-up lens. Specifically, firstly, intrinsic parameter calibration is performed on the panoramic lens and each close-up lens to obtain their internal parameters, including focal length, principal point coordinates, and lens distortion coefficients. Simultaneously, through multi-view joint calibration, the extrinsic parameters of each lens in a unified world coordinate system are determined, namely their respective rotation matrices and translation vectors. Based on the above intrinsic and extrinsic parameters, a complete projection model from the image plane to the world coordinate system can be constructed for each lens. On this basis, by assuming that the target point is located on a known physical reference plane (such as the ground), the coordinates in a certain lens image can be back-projected to world coordinates, and then reprojected onto its image plane through the projection model of the target lens, thereby achieving high-precision coordinate transformation across lenses. This implementation uses a physical imaging model for coordinate transformation, which can provide more accurate mapping results when dealing with scenes with large angle differences or significant lens distortion.

[0048] Step S30: For the attendees who fall into the multiple close-up shots, determine the target close-up shot from the set of close-up shots based on the imaging quality of the attendees in each close-up shot.

[0049] It should be noted that image quality refers to the integrity of the composition of the person in the close-up shot and the clarity of the image. For example, whether the distance between the key points of the body (such as the top of the head, shoulders, and chin) and the edge of the frame meets the preset safety threshold to avoid the person being cropped, and the clarity of the attendee in the close-up image; the target close-up shot refers to the selected shot that is ultimately used to output the close-up shot of the attendee.

[0050] As one possible implementation, step S30 includes steps S310 to S330: Step S310: Obtain the currently output video frame of each of the close-up shots in the close-up shot set.

[0051] In this embodiment, once it is determined that multiple close-up shots simultaneously capture the same attendee, an optimal shot must be selected from the set of close-up shots for subsequent tracking and image output to avoid identity duplication, resource redundancy, or image conflicts caused by the simultaneous use of multiple shots. Specifically, the video frames captured by each close-up shot in the set at the current moment are first acquired synchronously. To ensure consistent evaluation benchmarks, a timestamp alignment mechanism is used to extract image frames with the same time benchmark from the video stream buffers of each close-up shot; alternatively, a temporal alignment strategy based on frame sequence number matching or interpolation is used to select the effective frame with the smallest time difference as a replacement.

[0052] Step S320: Based on the video frames, calculate the imaging quality score of each close-up shot for the participants.

[0053] Step S330: The close-up shot with the highest imaging quality score is determined as the target close-up shot.

[0054] In this embodiment, after acquiring synchronized video frames, a deep learning-based image quality evaluation model is used to directly score the quality of the close-up images captured by each close-up shot through a trained convolutional neural network, and finally the target close-up shot is determined based on the quality score results.

[0055] Optionally, a human pose estimation algorithm (such as OpenPose) is used to extract the key points and coordinates of the participants in each close-up shot, and the image quality score is determined based on the spatial distribution of the key points. Specifically, firstly, it is detected whether any preset key points are missing; if any key point is missing, it is determined that the person in the close-up shot has a truncated phenomenon, and the composition integrity index is assigned a lower base score; if all key points are successfully detected, the composition quality is further quantified by calculating the vertical distance from the top of the head key point to the top edge of the close-up shot, the first horizontal distance from the left shoulder key point to the left edge of the close-up shot, and the second horizontal distance from the right shoulder key point to the right edge of the close-up shot. For the vertical distance, the first horizontal distance, and the second horizontal distance, they are compared with the corresponding preset boundary thresholds (e.g., 50 pixels), and the base score of the corresponding index is determined based on the comparison results. The indices referred to here are the three boundary distance indices: vertical distance, first horizontal distance, and second horizontal distance. Finally, the image quality score of the close-up shot for the participants is obtained by weighted fusion of the indices (i.e., composition integrity and the three boundary distance indices), and the close-up shot with the highest image quality score is selected as the target close-up shot.

[0056] To avoid frequent switching of close-up shots when their image quality scores are similar, a hysteresis threshold mechanism can be introduced, which only performs the switching when the image quality score of the new close-up shot is higher than that of the currently used close-up shot by a predetermined threshold.

[0057] Optionally, a minimum quality threshold can be set. If the image quality score of all close-up shots is lower than the minimum quality threshold, the panoramic image captured by the panoramic lens can be used as an alternative, from which the corresponding image of the attendee can be extracted.

[0058] It is understandable that multiple close-up shots may have overlapping fields of view, resulting in the same person being imaged in multiple close-up shots. However, some close-up shots may cause the person to be cut off due to skewed angles or excessive distance. Therefore, by conducting multi-dimensional image quality assessment, edge cropping issues can be effectively avoided, ensuring that the final output video stream has a complete composition, clear visuals, and meets image quality requirements.

[0059] Step S40: Based on the close-up shots of the participants output by the target close-up shot, generate a composite video image containing all participants.

[0060] It should be noted that composite video feed refers to integrating the independent display images of all participants (from close-up shots or cropped panoramic images) into a unified output screen according to a preset layout, forming a continuous, dynamically updated multi-channel video stream. The generation of this composite video feed is not a simple screen switching, but rather utilizes a parallel processing and synchronous output mechanism for multiple video streams. This mechanism periodically (e.g., every 200ms) tracks and extracts close-up shots of all participants in real time, dynamically aggregating them into a composite video stream containing all participants. This achieves the technical effect of real-time synchronous presentation and continuous tracking of multiple targets.

[0061] Close-up shots refer to high-quality local images in composite video footage that independently display each participant and meet the requirements of head and shoulder close-up composition.

[0062] To ensure that all participants in the composite video frame have a clear, independent display unit, the system can employ a dynamic adaptation strategy to generate close-up shots. Ideally, a single-person shot directly output from the target close-up, with a composition that meets the requirements, can be used directly as a close-up shot. However, in practical applications, due to the field of view and shooting angle of the close-up shot, a single frame may contain multiple participants simultaneously. In this case, after acquiring the original video stream output from the target close-up shot, image segmentation technology can be used to separate and extract independent close-up areas corresponding to different participants from the same shot. After cropping and scaling, a standard-compliant close-up shot is finally obtained, ensuring that each participant has an independent display unit.

[0063] In some implementations, if there is no effective close-up shot to cover the target attendee, the attendee's image can be cropped from the panoramic image as a temporary substitute. Although the image quality of the cropped image from the panoramic lens is usually lower than that from the close-up shot, it ensures the continuity of the attendee's display in the composite video frame.

[0064] Through the above multi-source adaptation strategy, the system can generate or adapt close-up images that conform to visual specifications for each participant based on real-time image conditions.

[0065] As a feasible implementation method, a deep learning-based person instance segmentation model (such as Mask R-CNN) can be used to process each frame of the close-up image. This model can simultaneously perform object detection and pixel-level segmentation, accurately identifying multiple human instances in the image and generating a corresponding binary mask for each instance. Then, by spatially matching the segmented human instances with the determined positions of attendees in the panoramic image, the mask region corresponding to the target attendee is determined. This specific mask region is then applied to the original video frame, and pixel-level matting can extract close-up images containing only the target attendee. Simultaneously, the background is processed to be transparent or a uniform color, effectively ensuring that even in high-density meeting scenarios, the close-up images of each attendee in the final composite video image have clear outlines and a professional visual effect.

[0066] As another feasible implementation, steps S401 to S403 are included before step S40: Step S401: Based on the facial recognition results of the close-up image, assign a corresponding identity identifier to each participant.

[0067] It should be noted that the identity identifier is used to continuously distinguish different individuals throughout the video conference, maintaining identity consistency even when they move between multiple cameras or are briefly obscured.

[0068] In this embodiment, a face recognition model is used to extract the feature vector of the face in the close-up image, and the similarity is compared with the face feature database. When the similarity exceeds a preset threshold, the close-up image is bound to the corresponding identity identifier.

[0069] Furthermore, upon acquiring a new close-up shot, the system generates or associates a corresponding identity identifier based on the facial recognition results within it. Specifically, during each frame processing, if the system detects that a participant in the close-up shot has not been identified in previous meeting proceedings (i.e., no existing identity identifier has been matched), it dynamically generates a new globally unique identity identifier for that participant and caches their facial features. If the participant has already been assigned an identity identifier, the current close-up shot is directly associated with that existing identity identifier, ensuring identity continuity across frames and shots. This implementation does not rely on pre-registration, supports impromptu meeting entry scenarios, and ensures the accuracy and stability of identity binding through feature caching and similarity verification.

[0070] Step S402: When acquiring the close-up image output by the target close-up shot, establish a mapping relationship between the identity identifier and the corresponding close-up image.

[0071] It should be noted that the target close-up refers to a close-up shot that is ultimately responsible for tracking a specific identity after panoramic detection, coordinate mapping, and multi-dimensional optimization.

[0072] In this embodiment, the system synchronously reads the video frames output from each target close-up shot and, in conjunction with the assigned identity identifiers, constructs a real-time updated mapping table. This table associates and binds each identity identifier with its currently corresponding close-up shot (or a display shot based on a cropped panoramic image). This mapping relationship is stored in memory as key-value pairs, where the key is the identity identifier and the value is the corresponding image data or image address.

[0073] Step S403: Based on the identity identifier and the mapping relationship, perform a deduplication operation on the close-up images associated with the same identity identifier to avoid the same participant being displayed repeatedly in the composite video image.

[0074] It should be noted that the deduplication operation refers to the process of ensuring that each identity has one and only one corresponding video source that is sampled and synthesized into the composite video when synthesizing the final composite video, based on the current mapping relationship.

[0075] In this embodiment, because multiple close-up shots may have overlapping fields of view, or because differences in processing timing may cause the same participant to be simultaneously output by different close-up shots within a short period of time, their identity may be temporarily associated with multiple close-up shots. Therefore, the system performs identity deduplication checks on all frames to be stitched together before stitching. If two or more frames are found to be bound to the same identity, only the most recently generated or best-quality frame is retained, and the rest are discarded.

[0076] In practice, a temporary composite list can be maintained. Before adding each frame, it is checked whether the identity identifier already exists in the list. If it already exists, the current frame is skipped or replaced with a better version. In this way, even in complex multi-camera environments, it can be ensured that each participant appears only once in the final output composite video, effectively avoiding the visual redundancy problem of "one person, multiple windows," and improving the professionalism of the image and the viewing experience.

[0077] Optionally, the system can adapt the screen arrangement according to a preset layout (such as a fixed layout of 2×2, 3×3 grid, etc.) or according to the actual physical seating arrangement in the conference room. It can synchronously align and stitch together the independent screens of all participants to form composite video frames at each moment. Based on this, it can continuously and periodically output smooth and stable composite video screens to ensure real-time synchronous updates and parallel display of the screens of all participants.

[0078] In this implementation, a unique identifier is assigned to each participant, and based on the correspondence between the identifier and the target close-up shot, a corresponding independent display frame is acquired in real time from multiple video streams. Subsequently, a mapping relationship between the identifier and the display frame is established, and multiple frames associated with the same identifier are deduplicated to ensure that each participant corresponds to only one valid frame at any given time. Finally, all valid frames are synchronously stitched together into a composite video frame, and a continuous video stream is periodically output. Compared to the traditional single-lens switching mode, which cannot simultaneously display close-up shots of multiple participants, this implementation achieves synchronous parallel display of close-up shots of all participants by acquiring multiple close-up video streams in parallel with multiple lenses, associating and binding close-up shots with identifiers, and performing dynamic aggregation and arrangement. While ensuring visibility for all participants, this, combined with an image quality optimization mechanism, significantly improves the image composition quality, visual integrity, and the immersiveness and interactive efficiency of remote meetings.

[0079] In summary, this embodiment acquires the global position information of participants through panoramic images. Combined with the pre-calibrated spatial mapping relationship between the panoramic lens and each close-up shot, it determines the set of close-up shots each participant falls into. Furthermore, for participants falling into multiple close-up shots, it selects the target close-up shot based on the image quality within each shot. Finally, based on the independent display screens of all participants, it generates a composite video stream containing all participants. This effectively solves the problems in traditional video conferencing systems, such as the inability of multiple participants to be presented synchronously and insufficient image resolution when shooting distant figures, caused by reliance on single-lens switching. It also avoids the technical problem of repeated images of participants appearing in the final screen when they are in the overlapping shooting areas of multiple cameras. This embodiment ensures high-quality output for each participant while achieving real-time synchronous visualization of all participants, significantly improving the immersive experience and multi-party interaction efficiency in remote collaboration and video conferencing scenarios.

[0080] Based on the above embodiments of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 In the synchronous display method of the video conference, step S20 includes steps S210 to S230: Step S210: The position information of the participants in the panoramic image is transformed to the image coordinate system of each of the close-up shots through the spatial mapping relationship to obtain the mapped coordinates of the participants in each of the close-up shots; wherein, the position information is the coordinates of the center point of the face detection box corresponding to the participants in the panoramic image, and the mapped coordinates are the coordinates of the center point after mapping in the image coordinate system of each of the close-up shots.

[0081] It should be noted that the face detection box refers to the rectangular bounding box corresponding to the facial region of the participant identified in the panoramic or close-up image by the object detection algorithm. Its geometric center point is used as the representative position of the participant in the picture to simplify cross-shot mapping calculation and maintain tracking stability.

[0082] In this embodiment, the aspect ratio coefficient of each close-up shot relative to the panoramic shot is obtained, and then the position information of the participants in the panoramic image is converted into the mapped coordinates in the image coordinate system of each close-up shot according to the aspect ratio coefficient.

[0083] To address the effects of lens distortion and perspective shift, the system assigns corresponding aspect ratio coefficients to different imaging areas of each close-up shot, enabling more accurate coordinate mapping. In practice, the system first determines the region type of each participant based on their position in the panoramic image. Then, it uses the aspect ratio coefficient corresponding to that region for coordinate transformation, effectively eliminating local distortion errors caused by lens characteristics and improving the overall accuracy of coordinate mapping.

[0084] Step S220: Determine whether the attendee exists within the field of view of each of the close-up shots based on the mapped coordinates.

[0085] It should be noted that the field of view refers to the effective imaging area where a close-up shot can capture a clear image. In this embodiment, the field of view boundary parameters for each close-up shot are pre-stored, including the pixel ranges corresponding to the image width and image height, respectively. During the determination, the mapped coordinates are compared with the field of view boundary of the corresponding close-up shot. When the x-value of the mapped coordinates is between 0 and the image width, and the y-value is between 0 and the image height, it is determined that the attendee is within the field of view of that close-up shot.

[0086] Optionally, a safety boundary threshold is set to define the core imaging area of ​​the close-up shot. Specifically, based on the image width and height, a preset safety boundary threshold (e.g., 5% of the image width and height or a fixed pixel value) is shrunk inward to form a core imaging area smaller than the actual field of view. Only when the mapped coordinates are within the core imaging area is it determined that the participant is within the field of view of the corresponding close-up shot, thus avoiding potential imaging quality problems when the person is at the edge of the frame.

[0087] Step S230: Determine the set of close-up shots based on the determination results of whether the attendees are within the field of view of each close-up shot.

[0088] In this embodiment, the determination results of all close-up shots are iterated, and the numbers of all close-up shots corresponding to the "existence" state are added to the close-up shot set of the attendee. If the close-up shot set is empty, it means that no close-up shot can effectively cover the attendee at present, and the attendee will be cropped from the panoramic image captured by the panoramic lens.

[0089] This embodiment realizes the coordinate mapping from panoramic lens to close-up shot through spatial mapping relationship, and combines the effectiveness judgment of field of view range to accurately construct the close-up shot set corresponding to each participant, effectively avoiding lens misselection caused by invalid coverage; on this basis, it provides a reliable basis for subsequently selecting high-quality images from multiple close-up shots, thereby significantly improving the clarity of human images in composite video images while ensuring that all participants can see synchronously.

[0090] Based on the above embodiments of this application, in the third embodiment of this application, the same or similar content as the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 In the synchronous display method of the video conference, step S40 includes steps S410 to S450: Step S410: Obtain the actual coordinates of the attendee in the close-up image output by the target close-up shot.

[0091] It should be noted that the actual coordinates refer to the pixel position of the geometric center point of the bounding box of the face corresponding to the participant, obtained by performing face detection in the close-up shot, in the coordinate system of the close-up image.

[0092] In this embodiment, a target detection algorithm can be used to process the video frame currently output by the close-up shot of the target to detect whether there is a valid face region. If the detection is successful, the corresponding face bounding box is extracted, and the midpoint between its upper left and lower right corners is calculated as the actual coordinates of the participant in the close-up shot.

[0093] Step S420: Determine the global mapping coordinates of the participants in the panoramic image based on the actual coordinates and the spatial mapping relationship.

[0094] In this embodiment, the system establishes the spatial mapping relationship between each close-up shot and the panoramic shot during the calibration phase, and pre-stores the horizontal and vertical aspect ratio coefficients for each close-up shot. After obtaining the actual coordinates of the participants in the close-up shot, the estimated position of the actual coordinates in the panoramic image coordinate system can be calculated in reverse according to the aspect ratio coefficient corresponding to the close-up shot, that is, the global mapped coordinates, thereby realizing spatial alignment and coordinate mapping from the local view to the global image.

[0095] Step S430: Obtain the center coordinates of the face detection boxes corresponding to each participant in the panoramic image.

[0096] In this embodiment, the system synchronously performs global face detection on panoramic image frames with the same timestamp, outputs the bounding boxes of all visible attendees' faces using a target detection algorithm, and calculates the geometric center coordinates of each face bounding box, i.e., the center coordinates. These center coordinates constitute a panoramic coordinate set, used to characterize the spatial distribution of all attendees in the panoramic image at the current moment.

[0097] Step S440: Calculate the spatial distance between the global mapping coordinates and each of the center coordinates, and determine the face detection box with the smallest spatial distance as the target detection box corresponding to the close-up image.

[0098] In this embodiment, due to factors such as lens distortion or calibration errors, the global coordinates obtained by reverse mapping from the close-up image may have certain deviations. Therefore, it is necessary to verify them with the panoramic detection results to ensure matching accuracy. Specifically, by calculating the Euclidean distance between the global mapped coordinates and each center coordinate, the face detection box corresponding to the center coordinate with the smallest distance is selected as the matching target (i.e., the target detection box), and a correspondence is established between the target detection box and the current close-up image.

[0099] If the minimum distance exceeds a preset spatial tolerance threshold (e.g., 100 pixels), the match is deemed unreliable, and the close-up image will not be included in subsequent compositing to avoid identity mismatch or image misalignment. For attendees lacking close-up images, the corresponding images can be directly cropped from the panoramic image and included in the compositing.

[0100] Step S450: Generate the composite video frame based on the correspondence between the target detection box and the close-up image.

[0101] In this embodiment, based on the correspondence between the target detection box and the close-up image, each close-up image is bound to the identity or location of the participant confirmed in the panoramic image, ensuring that each participant in the panoramic image is associated with at most one valid close-up source. Subsequently, all associated and bound close-up images are collected and adaptively sorted according to their coordinate order in the panoramic image, or arranged in a fixed grid layout. The size of each close-up image is uniformly normalized and stitched to a unified output canvas to generate a composite video frame containing all participants. This process is executed at a fixed frame rate period, outputting a continuous and stable composite video stream.

[0102] Through the aforementioned cross-view coordinate mapping, matching verification, and unique binding mechanism, this embodiment effectively solves the problems of identity confusion and image redundancy caused by mapping drift or detection mismatch in multi-lens systems, and achieves high-precision and consistent synchronous display of multiple close-up images.

[0103] As one possible implementation, step S450 includes steps S4510 to S4520: Step S4510: For each face detection box in the panoramic image, if there is no close-up image associated with the face detection box, then based on the position of the face detection box in the panoramic image, the corresponding display image is cropped from the panoramic image.

[0104] Step S4520: Generate the composite video frame according to the correspondence between the target detection box, the close-up view, and the display view.

[0105] It should be noted that this implementation aims to construct a complete set of display sources to ensure that the composite video image can present every participant without omission. Its core logic is that when some participants are not covered by close-up shots, the corresponding image of the missing participant from the panoramic view can be used as a supplement.

[0106] Optionally, the system maintains a real-time updated binding list, which records the correspondence between successfully associated target detection boxes and close-up images. Before synthesizing each frame of composite video, the system iterates through all face detection boxes in the current panoramic image and queries the binding list: if a face detection box in the panoramic image does not find a corresponding close-up image in the binding list, it is determined that the attendee currently lacks a valid close-up video source. Subsequently, based on the position and size of the face detection box, the system defines a cropping region in the panoramic image according to a preset scaling ratio (e.g., expanding the bounding box to 1.5 times its original size in both width and height to include more shoulder and upper body information), and extracts the image content within the cropping region as an alternative display image for the attendee.

[0107] Finally, based on the correspondence between the target detection boxes and the close-up and display screens, all close-up screens and alternative display screens obtained by cropping panoramic images are integrated. According to the position information of their respective target detection boxes, these screens are organized according to preset rules (such as row arrangement, grid layout, etc.) to form a composite video frame containing all participants, and finally a composite video screen is obtained.

[0108] To ensure visual continuity and harmony, necessary post-processing operations such as size adjustments and color corrections can be performed on each input screen. Furthermore, the overall layout of the composite video screen can be optimized, for example, ensuring that the relative positions of adjacent participants closely match their distribution in an actual meeting setting, thereby enhancing the realism and immersion of the viewing experience. Finally, the entire process will run continuously at a fixed frame rate, outputting a stable and smooth composite video stream, achieving efficient and accurate multi-screen synchronous display.

[0109] As another feasible implementation, steps S460-S470 are included after step S450: Step S460: When the participant is detected to have moved out of the field of view of the current target close-up shot, a new target close-up shot that the participant is about to enter is predicted based on the participant's motion trajectory in the previous tracking cycle.

[0110] In this embodiment, the system continuously monitors whether the mapped coordinates of each participant in the close-up shot are close to or beyond the imaging field of view boundary in each processing cycle. Once it is determined that a participant is about to move out of the field of view of the current target close-up shot (for example, its mapped coordinates are getting closer and closer to the imaging field of view boundary for several consecutive frames), the system fits the participant's movement direction and speed according to the participant's historical position sequence in the panoramic image coordinate system, and predicts the adjacent close-up shot that it is most likely to enter by combining the coverage area layout of each close-up shot in the panoramic image. The prediction result is used to activate or prepare the corresponding video stream in advance, providing a basis for smooth switching.

[0111] Step S470: Switch the close-up video stream corresponding to the participant from the current target close-up to the new target close-up, and continue to generate the composite video image based on the close-up image output by the new target close-up after the switch.

[0112] In this embodiment, based on the prediction results, the participant's identity and subsequent image binding are migrated from the original close-up shot to a new target close-up shot. From the next processing cycle onwards, the close-up image output in the new close-up shot is used as the effective video source to participate in subsequent identity mapping, deduplication, and composite video image generation processes. A brief transition buffer mechanism can be introduced during the switching process (such as retaining the previous frame until the new image stabilizes) to avoid image jumps or identity loss due to camera switching, thereby ensuring the continuity and stability of the composite video output.

[0113] For example, to help understand the implementation process of the video conferencing synchronization display method obtained by combining this embodiment with the above embodiments, please refer to... Figure 5 , Figure 5 A simplified flowchart of a synchronous display method for video conferencing is provided, specifically: First, the panoramic lens and multiple close-up lenses deployed in the conference scene are jointly calibrated. The spatial mapping relationship between each close-up lens and the panoramic image is established through homography transformation. The aspect ratio coefficient of each close-up lens relative to the panoramic image in the horizontal and vertical directions is calculated, which provides a basis for subsequent cross-view coordinate transformation.

[0114] During the meeting, the system acquires panoramic images in real time and performs object detection on the panoramic images to obtain the face bounding boxes of all participants and their geometric center coordinates. Based on the center coordinates and the aforementioned spatial mapping relationship, it determines which close-up shots each participant falls within the field of view, thereby determining the set of close-up shots corresponding to each participant.

[0115] For attendees who fall into multiple close-up shots, the imaging quality score of each video frame is calculated by reading the current output video frames of each close-up shot in the close-up shot set (e.g., based on indicators such as sharpness, brightness, and face ratio). The close-up shot with the highest imaging quality score is determined as the target close-up shot of the attendee, and its output video frame is used as the close-up shot of the attendee.

[0116] If a participant is not within the field of view of any close-up shot, an alternative display image containing the head and shoulders is cropped out using the face detection box in the panoramic image as the center and at a preset scaling ratio (e.g., 1.5 times).

[0117] The system then binds each participant to their uniquely selected close-up shot (or an alternative display shot cropped from a panoramic image) and maintains a real-time updated binding list. Before generating the composite video shot, a deduplication check is performed to ensure that the same participant does not appear repeatedly due to multiple camera coverage.

[0118] Finally, the system arranges all participants and their corresponding close-up shots according to their horizontal position in the panoramic image or a preset grid layout. After uniformizing and normalizing the size, the images are stitched together into a single-frame composite video image and continuously output at a fixed frame rate to form a synchronized video stream containing all participants and with clear images.

[0119] This application provides a synchronous display device for video conferencing, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the synchronous display method for video conferencing described in Embodiment 1 above.

[0120] The following is for reference. Figure 6 This illustration shows a schematic diagram of a synchronous display device suitable for implementing video conferencing embodiments of this application. The synchronous display device for video conferencing in these embodiments may include various hardware and software components for implementing synchronous display methods for video conferencing. Figure 6 The video conferencing synchronization display device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0121] like Figure 6As shown, the synchronous display device for video conferencing may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the synchronous display device for video conferencing. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the synchronous display device for video conferencing to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows synchronous display devices for video conferencing with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0122] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0123] The synchronous display device for video conferencing provided in this application, employing the synchronous display method for video conferencing in the above embodiments, can solve the technical problem of the same participant being repeatedly displayed in a composite image due to cross-shooting by multiple cameras. Compared with the prior art, the beneficial effects of the synchronous display device for video conferencing provided in this application are the same as those of the synchronous display method for video conferencing provided in the above embodiments, and other technical features of this synchronous display device for video conferencing are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0124] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0125] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0126] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the video conferencing synchronization display method described in the above embodiments.

[0127] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0128] The aforementioned computer-readable storage medium may be included in the synchronous display device of the video conferencing; or it may exist independently and not be assembled into the synchronous display device of the video conferencing.

[0129] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a video conferencing synchronization display device, cause the video conferencing synchronization display device to: determine the position information of each participant in the panoramic image captured by the panoramic lens; determine the set of close-up shots in which each participant falls based on the position information and the spatial mapping relationship between the panoramic lens and each of the close-up shots; for participants falling in multiple close-up shots, determine a target close-up shot from the set of close-up shots based on the imaging quality of the participant in each of the close-up shots; and generate a composite video image containing all participants based on the close-up shots of the participants output by the target close-up shots.

[0130] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0132] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0133] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the synchronous display method of the video conference described above. This solves the technical problem of the same participant being repeatedly displayed in a composite image due to multiple cameras capturing images simultaneously. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the synchronous display method of the video conference provided in the above embodiments, and will not be elaborated upon here.

[0134] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video conferencing synchronous display method described above.

[0135] The computer program product provided in this application can solve the technical problem of the same participant being repeatedly displayed in a composite image due to multiple cameras shooting simultaneously. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the synchronous display method for video conferencing provided in the above embodiments, and will not be repeated here.

[0136] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0139] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for synchronous display in a video conference, characterized in that, Applied to a video conferencing system, the video conferencing system includes a panoramic lens and multiple close-up lenses, and the synchronous display method of the video conferencing includes: Based on the panoramic images captured by the panoramic lens, determine the position information of each participant in the panoramic images; Based on the location information and the spatial mapping relationship between the panoramic lens and each of the close-up shots, the set of close-up shots in which each participant falls is determined; For the attendees who fall into multiple close-up shots, a target close-up shot is determined from the set of close-up shots based on the imaging quality of the attendees in each close-up shot; Based on the close-up shots of the participants output from the target close-up shots, a composite video image containing all participants is generated.

2. The synchronous display method for video conferencing as described in claim 1, characterized in that, The step of determining the set of close-up shots in which each participant falls, based on the location information and the spatial mapping relationship between the panoramic lens and each of the close-up shots, includes: The position information of the participants in the panoramic image is transformed to the image coordinate system of each of the close-up shots through the spatial mapping relationship to obtain the mapped coordinates of the participants in each of the close-up shots; Wherein, the location information is the coordinates of the center point of the face detection box corresponding to the participant in the panoramic image, and the mapped coordinates are the coordinates of the center point after mapping in the image coordinate system of each close-up shot; Based on the mapped coordinates, determine whether the attendee is within the field of view of each of the close-up shots; The set of close-up shots is determined based on the determination results of whether the attendees are within the field of view of each close-up shot.

3. The synchronous display method for video conferencing as described in claim 1, characterized in that, Before the step of determining the set of close-up shots in which each participant falls based on the location information and the spatial mapping relationship between the panoramic lens and each of the close-up shots, the synchronous display method of the video conference further includes: The panoramic lens and each of the close-up lenses respectively acquire calibration images containing multiple calibration reference points; Based on the corresponding image coordinates of each calibration reference point in the calibration image in the panoramic lens and each of the close-up shots, calculate the homography transformation matrix between the panoramic lens and each of the close-up shots; Based on the homography transformation matrix, the aspect ratio coefficient of each close-up shot relative to the panoramic shot is determined; Based on the homography transformation matrix and the image scaling factor, a spatial mapping relationship is established between the panoramic lens and each of the close-up shots.

4. The synchronous display method for video conferencing as described in claim 1, characterized in that, The step of determining a target close-up shot from the set of close-up shots based on the image quality of the attendee in each of the multiple close-up shots includes: Obtain the currently output video frame of each of the close-up shots in the set of close-up shots; Based on the video frames, calculate the image quality score for each close-up shot of the attendee; The close-up shot with the highest image quality score is identified as the target close-up shot.

5. The synchronous display method for video conferencing as described in claim 1, characterized in that, Before the step of generating a composite video image containing all participants based on the close-up shots of the participants output from the target close-up shots, the synchronous display method of the video conference further includes: Based on the facial recognition results of the close-up images, a corresponding identity identifier is assigned to each of the attendees; When acquiring the close-up image output from the target close-up shot, a mapping relationship is established between the identity identifier and the corresponding close-up image; Based on the identity identifier and the mapping relationship, the close-up images associated with the same identity identifier are deduplicated to avoid the same participant being displayed repeatedly in the composite video image.

6. The synchronous display method for video conferencing as described in claim 1, characterized in that, The step of generating a composite video image containing all participants based on the close-up shots of the participants output from the target close-up shots includes: Obtain the actual coordinates of the participants in the close-up image output by the target close-up shot; Based on the actual coordinates and the spatial mapping relationship, determine the global mapping coordinates of the participants in the panoramic image; Obtain the center coordinates of the face detection bounding boxes corresponding to each of the attendees in the panoramic image; Calculate the spatial distance between the global mapping coordinates and each of the center coordinates, and determine the face detection box with the smallest spatial distance as the target detection box corresponding to the close-up image; The composite video frame is generated based on the correspondence between the target detection box and the close-up image.

7. The synchronous display method for video conferencing as described in claim 6, characterized in that, The step of generating the composite video frame based on the correspondence between the target detection box and the close-up shot includes: For each face detection box in the panoramic image, if there is no close-up image associated with the face detection box, then based on the position of the face detection box in the panoramic image, the corresponding display image is cropped from the panoramic image. The composite video frame is generated based on the correspondence between the target detection box, the close-up shot, and the display shot.

8. The synchronous display method for video conferencing as described in claim 6, characterized in that, After the step of generating the composite video image based on the correspondence between the target detection box and the close-up image, the synchronous display method of the video conference further includes: When the participant is detected to have moved out of the field of view of the current target close-up, the new target close-up that the participant will enter is predicted based on the participant's movement trajectory in the previous tracking cycle. The close-up video stream corresponding to the attendee is switched from the current target close-up to the new target close-up, and the composite video image is generated based on the close-up image output by the new target close-up after the switch.

9. A synchronous display device for video conferencing, characterized in that, The synchronous display device for the video conference includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the synchronous display method for the video conference as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the synchronous display method for video conferencing as described in any one of claims 1 to 8.

Images